I am absolutely new to AI/ML and need some guidance/direction.

Every “New to AI, try this” guide I find ends up going down a path that isn’t right for the project I’m working on - or convoluted with so many terms I need to look up, I get rather frustrated. Maybe I’m too old to learn/use AI? Anyway . . .

This is my project, and any guidance, pointers, help would be super appreciated. I’m working on a job aggregator. I have a simple web crawler that goes to a url, fetches the HTML, cleans a lot of the text and structure, and outputs the content of the job posting.

I then go in manually, look at that simplified HTML and extract the actual job description (vs Company description, benefits, other stuff on a job posting) to be used in another database. I use the exact wording, straight copy and paste, no summarization or interpretation.

I have about 400 data points in a database that look like this: job_site: “COMPANY_NAME”, raw_html: “<h1>Job Title</h1><p>This is what we do</p><p>We are looking for someone who</p>” job_description: “We are looking for someone who” That I’ve manually extracted. I feel like I can use that as training data to do some form of text . . . extraction ?? . . . from an html document. But I don’t have any clue on where to start

  • @LoopedcandleOPEnglish
    2•1 year ago

    Thanks for this! I’ll start learning!

    A friend mentioned I should start with a pre-trained model because 400 (and growing 50ish / week with my crawler) is just not nearly enough. Then do continued learning on that pre-trained model. Does that sound right?

    • @coolkicksEnglish
      1•1 year ago

      Yeah, model training is hard. Like capital H HARD. you need a bunch of data and it needs to be high quality.

      New York is the financial center of USA, so separating finance jobs from job postings written by someone using New England vernacular is a step you need to go through to make sure your data is high enough quality.

      So if you are just starting, use 20 newsgroups dataset in those links, it’s pretty good data with a ton of resources written about it. It’s not fun data, but it isn’t as likely to fall victim to biases in data you aren’t expecting.