Board 8 > I just found out something about the lurker site

(+) Yeah! (+)

okaaaaaaay back to doing stuff and I've pseudo-coded a solution that seems fine... but now to actually translate it to the real thing which augh hurts my head to even think about
---

how come users like Full Throttle get their topics in but have blank posts
---

th3l3fty posted...

how come users like Full Throttle get their topics in but have blank posts

polls

known issue with a quick fix for a while but I don't plan on getting to it until I fix the database builder!

oh and lots of topics will probably be messed up from a couple days ago where the board scanner broke during a run, so topics were collected but posts weren't
---

ah

I thought it was some weird ignore list quirk at first
---

there is a matter of deleted messages not being accounted for, the database just skips over deleted posts instead of adding a dummy so some pages won't match up for that reason
---

and so I start my first full test

kinda want to sleep now
---

welp everything's set back up again, this time just gonna re-scan b8 ever 8 hours (at least for now) and see how that fairs through the rest of the week
---

ps god I hate shell scripting
---

so I fixed the display of poll OPs but unfortunately the issue was in the actually post reader so I can't fix topics retroactively, only poll topics read from now on will be fixed

I do enjoy the style I got for 'em though!

http://gmun.moe/b8/lurker/topic.php?id=75142002

... yeahhh I don't save the View Results versions which I suppose would be easy to change (... I'd just have to add &result=1 to all the pages being read... which would require other smaller tweaks) buuut I don't re-scan old posts anyways and I don't plan on changing that so it doesn't really matter, since results would all be set in stone from the first time the topic is scanned
---

did ulti sue those people yet
---

Punnyz posted...

did ulti sue those people yet

Yeah but he only got 500k
---

Are you actively archiving these topics?
---

Actively?

I gotta sleep so I'll just give some in-depth explanation to cover your question. The whole setup works in 3 separate "parts":

- a MySQL database to store topics/posts
- a set of shell & python scripts to read Board 8, parse it, and upload topics/posts to the MySQL database
- lurker site pages read the MySQL database through php and displays it neatly & styled as any webpage would.

Each of these parts are on my web server. They don't necessary have to be... but uh it's simple. For the shell/python scripts, since my server's in Linux I just have a crontab scheduler that runs the master script each day at 0:00, 3:00, 6:00... and so on.

I *can* update manually on my comp (and I did earlier when fixing up polls) but uh it's more reliable to just let my server do it since it's always up 'n running. Actually I was writing up the new version on an Amazon (AWS) instance just-in-case so there's always the option to move back over there.
---

i'm just wondering if your thing is archiving these posts. For example if this topic purged today would it have been kept? What about other topics?
---

GuessMyUserName posted...

welp everything's set back up again, this time just gonna re-scan b8 ever 8 hours

It updates itself every 8 hours so nothing can purge before an update.
---

You can look at it and see it still has topics from two weeks ago.
---

GiftedACIII posted...

i'm just wondering if your thing is archiving these posts. For example if this topic purged today would it have been kept? What about other topics?

The topic is put on my database so yes, it will always stay there.

However... if you'll notice it currently broke again. GameFAQs blocked out my server from dl'ing b8 pages so I gotta offload those tasks somewhere else now.
---

How many topics are you archiving?
And for later reference, this site purges at 7:00AM est (so in 4 hours 30 minutes from now) and they take out everything from 8:00PM.
---

There are currently 748 topics in the database. Literally everything that gets posted here gets archived. (at least... whenever I fix up the recent roadblock)

For reference, B8 right how has 258 active topics
---

I think maybe he thinks you're archiving the topics manually?
---

yeahhh that was my first thought but I tried to cover that question!

GuessMyUserName posted...

I just have a crontab scheduler that runs the master script each day at 0:00, 3:00, 6:00... and so on.

^

master script runs through every page of b8 looking for topics that either aren't in the archive, or with an outdated # of "Last Posts", and adds 'em all




annnnyhow I got this running on a separate instance, doing the first full scan now... again......... gonna a take some time since it's been a while since the last update T_T
---

annnnd we're good, hopefully at every 12 hours this time won't have a problem!

only time will tell!
---

nice. Do you have the link to it?
---

...? I thought you've been following the page all this time!

http://www.gmun.moe/b8/Lurker

at least that's the link typing from memory, I'm out right now so yeah

although roadblocked by gamefaqs again so I'll have to fix something... again... later
---

lol sorry, I was posting right before I went to bed and I just wanted to find an excuse to bump this. So are the topics going to be archived there permanently? Or will they eventually purge too?
---

well for as long as I'm hosting my site!... which I guess could be as long as I live because I use it for all sorts of my day-to-day things

but urgh if gamefaqs roadblocks still comin', I'll try going one step further on something...

Final-Fantasy-6 posted...

well for as long as I'm hosting my site!... which I guess could be as long as I live because I use it for all sorts of my day-to-day things

but urgh if gamefaqs roadblocks still comin', I'll try going one step further on something...

Nice, nice. I'm assuming you're GMUN? And I just realized the domain of that website is the acronym of your name lol
---

yar~... I have many alts and i just figure to make use of 'em these days

lol I hope the roadblocks aren't too troublesome for you
---

Once I get things completely sorted out I should be able to leave it alone buuuut in the meantime it's still taking active work

The thing is if i take too long to update now, the next run will go through so many pages that GameFAQs bans that "computer" before it even finishes. However, if I have a single "computer" running the scan too frequently, while making the individual scans smaller, it will still make that "computer" easily detected just the same. I could schedule a whole bunch of different "computers" to alternately scan at frequent intervals so that each scan is quick and undetected, and no single device is running again for a long time... but that would take a lot of devices to set up so I'm just figuring out what I can get away with.

It also helps to schedule around B8's more active traffic periods -- if there's no new posts made for a while then scans will finish without going through a lot of pages so I can space them out easier.

I could probably make things easier if I log into an account with 50 ppp/tpp... but ofc I'm a bit wary of having something like that on my usermap........ although hrmmmmmm, I can do something about that...... we'll see
---

hrmmmmmmmmmm so I got the user thing going with 50 ppp/tpp which should make things better...... but now since the scanner is logged into an account the http formatting is out of whack and it seems inconsistently so since my first fix didn't solv-........... ohwait if I look at the url of their error'd topics I get it. I solve one thing but a different version of the same problem occurs. Maybe.

..... either way I'm getting sleepy so if it takes too much I'll probably leave it for tomorrow



ps lmao the account got RO'd

https://www.gamefaqs.com/community/B8Lurker/bio

... but hey at least personalized "Read"ing is the "Only" thing I need!
---

GuessMyUserName posted...

Once I get things completely sorted out I should be able to leave it alone buuuut in the meantime it's still taking active work

The thing is if i take too long to update now, the next run will go through so many pages that GameFAQs bans that "computer" before it even finishes. However, if I have a single "computer" running the scan too frequently, while making the individual scans smaller, it will still make that "computer" easily detected just the same. I could schedule a whole bunch of different "computers" to alternately scan at frequent intervals so that each scan is quick and undetected, and no single device is running again for a long time... but that would take a lot of devices to set up so I'm just figuring out what I can get away with.

It also helps to schedule around B8's more active traffic periods -- if there's no new posts made for a while then scans will finish without going through a lot of pages so I can space them out easier.

I could probably make things easier if I log into an account with 50 ppp/tpp... but ofc I'm a bit wary of having something like that on my usermap........ although hrmmmmmm, I can do something about that...... we'll see

There must be a way for the site not to automatically attack you. I mean, the original L site managed to slink by unnoticed for quite awhile. hmm.
---

WHY ARE YALL STILL USIN MY TOPIC FOR THIS?!?!

caps
---

Why make another topic when this one is in perfectly good working order?
---

so at someone's request...

http://www.logfaqs.com/

I'd still have to move over the data from the old database (and... adjust some things since I changed a bit)... but I'm also tempted to write something up to pull from the old lfaqs website.
---

and I am now in the process of absorbing the old lfaqs

unfortunately they never saved post IDs, sooooo.... I'm just sticking all lfaqs posts to their own ID together

http://logfaqs.com/post.php?id=10

... not exactly sure why topic titles don't work on them on this page specifically but whatever



also wtf the "Punnyz" name-change is 5 years old now?!
---

The funniest thing about this is theres a 30% chance sballen is reading this topic and shaking his head
---

tbqh I'm terrified whenever this topic comes back up to the front page


but my current crawler is a nice bot now! instead of faking things I'm just straight-up identifying myself, with info/contact link that goes to a page whining about the purge rate causing my old husbando ranking topic to die and lose my write-ups

I've actually learned a bunch doing this either way so were see

now I can probably fight off the russian bots that plagued my own board with porn
---

Hooray for learning things!

SantaRPidgey posted...

sballen

I read this as snalien and was wondering why he would care.
---

I like how youtube links are just pictures and not the link or the video
---

Huh GMUN your archive makes it seem like deleted posts never even happened.

That's an interesting way to handle it.
---

th3l3fty posted...

I like how youtube links are just pictures and not the link or the video

... I didn't notice but hey I could probably fix this since the img urls contain the video's ID string still. But I have a class to head off to now!

profDEADPOOL posted...

Huh GMUN your archive makes it seem like deleted posts never even happened.

That's an interesting way to handle it.

Yeahhhhh it's the only way to really handle 'em easily. No date or or post# not store for them so unless I put a deal of effort I wouldn't be able to have a dummy deleted post stored at the right position in the "topic".
---

GuessMyUserName posted...

Yeahhhhh it's the only way to really handle 'em easily. No date or or post# not store for them so unless I put a deal of effort I wouldn't be able to have a dummy deleted post stored at the right position in the "topic".

Fair enough.
---

dat feel when you accidentally use 'rm' in the parent directory instead of the child
---

I don't know that feel.
---

well I accidentally wiped out some big scripts I wrote

thankfully I was only in the folder with the scripts backing up the old lfaqs site and that I at least an acceptable backup of one of the files which I can reasonably adjust for the second.

so set that back up and going through lfaqs again



... but god damn this is a lot of data, my sql insertions are now at a crawl's pace
---