For instance, the gravity falls episode “society of the blind eye”, in the fandom.com wiki, there’s a synopsis for the episode. It would be incredibly helpful if I had some kind of generator, where I could paste the URL of said wiki page, and it would output a lorebook text file for me, with what the AI could extrapolate and phrase into said lorebook. Is this possible? have I somehow missed this generator? Please help, I’m sure someone’s thought of the need for this by now.
What you’re asking for is an automation tool called a “web crawler” or “spider” (IT guys love their jokes). Just a heads up, websites have a million reasons why they don’t like such tools and will very likely block your ability to do it easily.
Here’s one for Wikipedia. No clue if it’s baked in for just the main site or if you can use other wikis: https://janus-tg.github.io/Araneae/
If you want to do it right, learn python and have fun! There are scripts to pull web text contents, then you can truncate the parts you don’t want like the header, page tabs, and reference footer using string-parsing functions.
I’m currently in the process of making a script that gets every available URL from a website. It’s been done a hundred times before, but I’m doing it for learning.
OP, you should look into BeautifulSoup for Python. It would be a starting point for you. That will give you the full HTML of the webpage.
I tested your link with a character that can “read” websites to see how big the text actually is.
The highlighted part basically says “This is way too much text, I’ll ignore what is over the maximum I can handle.”
Since all of perchance uses the same model for AI text things, a generator with the sole purpose of creating lore out of something like that will face the same problem - you’d need to use a major company LLM for that with a higher token limit.This is ALL of the text on this site (~21,600 characters), just the synopsis text would be fine, that’s not even 7,000 characters.
I fed that synopsis to Loreena, a character created by Petra to create Lore entries, this is the result:

Maybe this is close enough to what you are seeking. If not, you can try the same with Gemini or other major company LLM for probably better results.




