DilloFAQs > Testing a new framework deployment of the site

Been working on a thing again for the past few days, got enough of it ready to deploy a public demo of what's available.

I originally made this site in a mess of terrible PHP duck-taped together like 6 years ago when I had much less experience than I do now. Since then with learning a whole bunch of frameworks I really wanted transition this place to use one.

... unfortunately with my hosting plan my options are severely limited, I've had a great time with JS frameworks (Express / React / Next, in the last thread I mentioned a bit) but that's a no-go here. I was then looking into Django which I thought I remembered having support but lol whoops nope there went that studying.

However! What I do know is supposedly supported is Flask... which I had a bad experience working with a long while ago, but now with experiencing some other frameworks I'm willing to give it another go and definitely implement better structural concepts.

https://logfaqs.com/omega/

So I got enough of it now to justify a public deployment, just finished configuring it into production and here it is. Obviously not fully-featured yet, I only so far implement basic browsing mechanics: home / board menu, topic list, post list, and post page, with page navigation.

Only the main / current database is linked up, so if you go to older topics or the manual board or this one you won't find anything yet. That being said I think it's looking much better with responsive design to adjust for mobile users finally... currently a bit of an issue with topic pages being themed with a darker pallet than the home page / topic list but both were looking nice while I was making them and I haven't normalized it yet but I'll work on it.

Deploying because I actually don't know if my hosting provider can actually handle it, like I said I tried Flask on this before and it had crashing problems so that's why I wanna test and see how it runs on-server. Worst-case scenario I'll have to rebuild my PHP infrastructure with what I've learned over the years but yikes ugh, maybe I'd look at other options tbh.
---

oh i probably shouldn't be populating a complete drop-down menu of the topic list that can have thousands of pages on it

... do i sleep or do i fix this first
---

I made the choice I always do - lmao sleep

tweaked it so for now the drop-down is restricted to +/-15 from the current page

probably best to have a gradual ramp-up like show 1-10, then 20 30 40 50, then 100 250 500 1000 10000. damn CE being so spread out across a gazillion different topics
---

Hard to remember all changes for today but Topic Search is up (just titles), connected the full list of databases so all topics are now viewable, and I think yesterday is when I normalized the theme colours.

Added back the little dillo icons on the topic list, but I've also now tweaked it so it will actually be useful - differentiating which topics have polls or not. Poll topics now use the red icon, lost topics use black.

Prepared the post search bar, but it doesn't do anything yet. Topic searching at least uses the same page as the topic list just like GameFAQs does, but post searches will require a unique page which I haven't yet decided on the specifics.

Hitting midnight so that's probably it for today... low-key I'd love to implement these boards here so I can post these updates through the new design instead of this ugly place. But that'll require a few updates like handling sessions to store login state and of course the login page. So next is probably post search.
---

maybe the header drop-down menus lol

oh yeah I still don't even have the hamburger menu working for mobile whoops, but it doesn't matter yet since those topnav links don't even matter yet
---

I lied about sleep and kept working again

let's see uhhh Blacklist page finished up since that was simple, and then just a bunch of work on the top nav bars

Boards & Yeahs have a drop-menu again (although Yeahs don't go anywhere yet), and then I fixed the mobile hamburger menu

not fully yet, mostly just the act of getting it to appear on click, adjusting the theme for when the hamburger menu pops up, and closing

oh yeah and i took away the x margins on the board/topic list so everything goes full-width when on mobile now.

tomorrow finish up tweaking hamburger menu then get on that posts search
---

So I added view functionality to the import boards & this one... NOW post search is next on the list. Just gonna note a little checklist here now to see what's left.

Post Search
User Page
User Topic Search
User Post Search
Fav'd Topics
Fav'd Posts
Account Functionality
Account Filtering on Fav pages
Site Board Posting Functionality

I'm pretty happy that I've never seen the beta page down outside of periods I'm actively tweaking it, so I think I'll be able to shift it to the root page once all the functionality that exists here is ported.
---

Post search is on. Can't remember what else i was working on it's kind of a blur at this point.

I still need to make a database selector for the post search function, other forms of browsing don't require it because the topic list is kept in one main log and it keeps track of which database its posts exist within - all topics can be searched at once, and when you enter a topic it automatically knows to retrieve from its corresponding database.

Posts on the other hand are what's spread across all the 11 so far databases (gonna need to add another again soon), so that's why a selector's needed. And no way in hell would I use a union across each database knowing how long that'd make a search cost.

I'd like to get a more legitimate database host but the plan I'd require would cost more than I can afford yet so that'll have to wait. Maybe I'll add a donation link when all's done.
---

User Page, User Topic Search, User Page Search

check check check, also noting those User Topics and User Posts pages are searchable which the current site doesn't allow for yet.

Once again, databases are selectable through URL parameters like &db=5 (valid values -1 to 10) but there's no html elements to interact with that value yet, like with the Post Search rn

Might work on those selectors next, might jump to the Yeahs pages, both shouldn't be particularly big deals. Taking a break now though, probably come back to it in the night... leaning on the selectors, better shot of me finishing that before sleep.
---

wrapping my mind around Memento was quite the distraction, but I needed the break

database selectors for the post search & user post search pages done

Fav'd Topics
Fav'd Posts
Account Functionality
Account Filtering on Fav pages
Site Board Posting Functionality

remaining list
---

forgot the general user page also depends on database selection so put one there too woop
---

Had to switch over to fixing the scan bot since GFAQs changed the way it hands out ads again and it broke the first post in every topic.

Ended up rebuilding the thing... again. Did it better than a few months ago, using more concepts I learned from building the Flask page lol.

Running a local scan now to fix the post #1s of everything that's still up... although as i type that i notice I didn't handle deleted posts whoop so just another second to correct that. Then gotta move the new scan code on-site and monitor that it's working fine.

Beta progress gonna have to wait another day.
---

Juggling work between the scanner and the viewer, mainly trouble-shooting changes with the scanner.

Having difficulty updating poll topics's first posts for their results and it's pretty weird. I have a version I can run and it will work but what's in the automated section isn't. It's not much lines but there's something different such that it only works correctly in one.

Anyways, for the viewer haven't had much work on it but I did just add the Fav'd Topics page. Leave the posts for tomorrow then it'll be just about handling accounts.
---

Hrm.

I just added a select statement to pull all topics with the None issue into the small debugging debugging code and I just watched it clear out 43/44 of the affected topics (the 44th purged).

I don't get it it's literally the same lines of code, this is just me accessing one function that does all the work in one goal for the sole task of updating this list of topics as opposed to the exact same function in-between scans.

*shrug* I guess I can just schedule this function to run after the scan. Also, strange issue, some poll-less topics are identified as having a poll for some reason... I don't understand how, whether a topic has a poll or not is defined only by an assignment on post #1 in a topic that checks if there is a div with id poll_results or not. Yet there's somehow topics marked Poll that don't have this? Huhhhh, maybe this is left-over from a previous check I had??
---

... I... i think I got it fixed now.

So immensely frustrating to figure out this problem, why it would work running in one environment but not the host environment, and after changing the way my debug fixing script works I discovered two things -- that GameFAQs is very inconsistent in how it displays its data, sometimes wrapping the message body text with a "msg_text" wrapper but sometimes just *not* doing that. But also, even when it was, the html parser is malinforming the data and when there'd be a topic it would end the entire table element after the poll vote div closed for some reason - so the actual message data was outside of the searched td element.

So with changing what I'm looking for and instead using the html5lib parser I've got a scan running now and everything seems to be going well. Posts are uploaded correctly, with sigs, with poll data, just all good now finally - and better yet polls are being scanned correctly the very first time so I don't even have to run a post-scan correction function.
---

Been a bit busy with moving again, but okay I definitely for reals fixed the poll updating. Boy fixing the scanner really side-tracked from view development...... and then oh whoop, the database filled up again and I had to had the 15th. Approaching that maximum limit of databases my host offers!

I did add the post likes page though, so now it's finally time for account implementation.
---

Update, you can log-in.

Can't do anything with logging in yet buuuuuut god damn was that some work because my hosting plan really limits my options so the situation is a bit convoluted - you need to generate a new password to work the beta systems because I can't do on Flask what I can do on PHP and vice-versa.

So what so you have to do?

https://logfaqs.com/omega/login

Well if you check the omega login page you'll see the Forgot Password bit, it'll lead you to a Forgot Generator page on the classic site. So if you still know your credentials you can put them there, click Generate, and you'll have your Forgot Key set in the field so you can just log-in using your credentials & the key and your details will be updated.

If you don't remember your password, as always you just have to PM me and I'll set on up for you to create a new password.

Account creation works fine on the new page.

Like I said though accounts don't really do anything aside from appear on the header and you can logout but yeah I guess posting here will be next.
---

Don't mind me just testing post route
---

Okay that worked so can I use this extra bit
---

One more with increments
---

well if you note the other topics you'll see posts & topics can be made on beta now

testing online with this post >_>
---

welp it's online
---

hrm but it didn't update topic latest date

welp I'm eating now
---

Update: user search added back to home page, user settings can be edited again, and I've included a new feature to streamline the website view... I'm pretty sure it only reflects mobile, since that's what the request I got was focused on. Does hide Yeahs though.
---

oh hey also the dillos board is updating properly, I actually didnt tweak that but I guess that was just fixed in local and when I uploaded the new features that old fix finally came in.

anyways forgot to mention though, while timezone is updateable in settings, it isnt actually USED just yet
---

grrrrrr broke again. fixed, but I just wanna post about it because it's such a stupid thing.

web host updated some system settings somewhere that makes the flask app no longer read the "utf8" charset which is what my flask's mysqldb connector is trying by default. error pointing me towards a system file Index.html about lacking 'utf' so I go look at the file and yeah I see a utf charset with alias utf-8 only (not 'utf8') so i guess that's what I gotta change

okay so lemme google about changing flask's charset setting... one answer being setting MYSQL_DATABASE_CHARSET but that doesn't work oh someone else says it's actually just MYSQL_CHARSET yay that is showing changes!......... in my error message.... now the error says utf-8 does not exist either so huh okay I guess aliases just don't actually work so I'll use the full name in my Index.xml which is utf8mb3 and bingo webpage loads up again

so the one line of code needed in my flask mysql config was just
MYSQL_CHARSET = 'utf8mb3'
and all's good again because for some annoying reason they couldn't just keep a working alias for 'utf8' which has been working for months
---

Just wanna note that posts made after 11/17 in topics made before 11/17 have not been archived yet. Any messages from topics that purged in between now and then are gone but there wasn't a whole lot yet. Today the 11/17 topics just purged while the 11/16 topics that also purged today still had every post.