Ancestors

Toot

Written by naught101 on 2025-01-30 at 13:32

Is there #AI/#ML work focused on extracting meaning from text?

My understanding is LLMs don't do this, they just map text into a numerical space, and use that to generate new text (e.g. there's no text <-> meaning mapping, except in so far as the numerical space approximates meaning, i.e. not much). Hence LLMs generate nonsense that sounds correct, maths that's wrong.

Wondering if there are models that include components with explicit focus on #meaning (I don't know what this would look like)

=> More informations about this toot | More toots from naught101@mastodon.social

Descendants

Written by Alexandre D'Hooge on 2025-01-30 at 17:46

@naught101 there is actually a notion of "meaning" in how LLMs treat text, it's encapsulated in the Embeddings at the very beginning of the network. While words are indeed converted into numbers, they are converted in numbers that are close to each other if they are used in similar cases or around similar words. It's not exactly "meaning" as we humans would identify it but I'd say that's close enough 🤷‍♂️

=> More informations about this toot | More toots from soubaboy@mastodon.zaclys.com

Written by naught101 on 2025-01-30 at 21:53

@soubaboy

Hrm. The wikipedia LLM page says:

These models acquire predictive power regarding syntax, semantics, and ontologies

In my understanding, syntax and ontologies could basically be generated via correlations with no embedded meaning. Semantics is the interesting part, but the reference for that sentence doesn't mention semantics.. It does mention conceptual knowledge graphs though, and suggests there are some models that can generate them. That kinda sounds like what I'm looking for

=> More informations about this toot | More toots from naught101@mastodon.social

Written by Alexandre D'Hooge on 2025-01-31 at 08:00

@naught101

Hi! Not sure about what the Wikipedia reference refers to, it sounds like LLMs that would be trained on very specific tasks.

Basically a LLM is some kind of GPT model that is trained at predicting the next word given some previous context (actually the next token but anyway). Once it's trained on enough data to make sentences that make sense, it can be finetuned on tasks like question answering, translation, rewriting etc.

Amazing blog post here:

https://jalammar.github.io/how-gpt3-works-visualizations-animations/

=> More informations about this toot | More toots from soubaboy@mastodon.zaclys.com