technical

Some problems with Danish and Wikidata lexemes

Posted on

 

Ordia-professortiltrædelsesforelæsning

Is is at all possible to describe natural languages in a structured way? There are many special cases and oddities of the Danish language one continuously discovers when entering Danish lexemes in Wikidata.

  1. What do we do with øl (beer). It can both be a neutrum and an common gender (utrum) word and the semantics between the two versions differ. In Wikidata they can both appear under the same lexemes, but how one then keeps track of that one form is associated with one sense and another form with another sense is less clear. Qualifiers may be brought to help. There are, however, to my knowledge no property that currently can be used to this.
  2. Is hænge (hang) one or two lexemes? There is a transitive and an intransitive version where there is a slight semantic difference. Den Danske Ordbog (DDS) has only one entry for hænge and then spend some words explaining the form-sense complexity.  Wikidata has currently L45348 and L45349.
  3. Is the “an” in “ankomme“, “ankomst”, “anslå”, “anvende”, etc. a dedicated prefix or should be regarded as the “an” adverb attached to a verb or a noun? The problem with regarding “an” as a prefix is that many other words that prefix to komst are adverbs: “bort”, “frem”, “ind”, “op”, “sammen”, “til” and these units does not look prefix’ish to me.
  4. It is sometimes not clear where a part of compound should linked to. For instance, tilskadekomst (injury) can be decomposed into “til”, “skade” and “komst”. The “til” could be regarded as a preposition or an adverb. For indeklima (indoor climate), inde could be the adverb inde or the adverb ind plus an -e- interfix.
  5. Should briller (glasses) and bukser (trousers) be plurale tantum? In common language briller and bukser are plurale tantum, but among professional sales persons you find the singular versions brille and buks. How would you indicate that? Note that compounds with the words may have the singular versions, e.g., bukseopslag and brilleetui.
  6. For singulare tantum/countable nouns, singular forms of lexemes may be so prevalent and plural forms so rare that it may be a question whether the word is singulare tantum or a countable noun, e.g., tillid (trust) may be found in rare instances as tillider, but does that make is a countable noun?
  7. What word is komst? Is it a suffix? Then what about the word genkomst, – it has the prefix “gen-” and the the suffix komst…, so where is the root? Maybe it is better to regard it as a part of a tyttebærord, where a word once recognized as an independent word has “lost its independence”. Komst has an entry in the old Danish dictionary, but not in newer Danish dictionaries.
  8. Following Grammatik over det Danske Sprog (GDS), some nouns have been added as “nexual nouns” or “innexual nouns”. The precise delineation of these classes are not clear, e.g., where would agent words such as woman, carpenter and cat be placed? The are not nexual, as far as I can see, but does that make them innexual? There is a range of Danish words where I am unsure: landskab (landscape), musikrum (music room), etc. So far I have left any annotation of such words out.
  9. Where do beslutte and similar words derive from? According to Den Danske Ordbog (DDS), it derives from middelnedertysk “besluten”, but could also be regarded as derived from a “be-” prefix and the verb “slutte”. It is only partially possible to represent both paths in the derivation in Wikidata.
  10. Wikidata has the “lexical category” field. For an affix it is not clear what the category should be. It could be affix, suffix/prefix, or perhaps something else?
  11. A particular class of words at the intersection of nouns and verbs are what has been term centaur. They might be inflections of verbs or they might be derivations from verbs to nouns. Examples are råben (shouting as a noun), løben (running) and skrigen (screaming). They do not seem to have any inflections themselves, so should they then be regarded as just an inflection of a verb and put in as a form under the corresponding verbal lexeme, e.g., råbe? On the other hand, DDS has råben as an independent entry and I also added råben as an independent lexeme in Wikidata. In Wikidata, this enable a more straightforward link to the synonym råberi.
  12. Which lexical category should we link compounds to? Some compounds may be analyzed to arise from a noun or a verb (or possibly other lexical categories), e.g., springvand has the parts spring and vand. It is not – at least to me – clear whether spring should be regarded as linked to the noun spring or to the root form of the verb springe.
  13. Should the s-genitive form of Danish nouns be recorded under forms? The naïve approach is to add the s-genitive forms doubling the amount of Danish noun forms. Modern linguists seem think (if I understand them correctly) that the appended “s” is enclitic and the s-genitive not a form, – much like in English where the ‘s genitive are not recorded as a form. For English the apostroph separates the s from the rest of the word, so there is is natural not to include the genitive form.
  14. Hjem, hjemme and hjemad are three words an possibly one, two or three lexemes. If they are three lexemes then how can we link them?
  15. When is a noun derived from a verb and not the other way around? It is particularly a question for (possible) root derivations, where the noun is shorter than the verb. For the noun bijob and the verb bijobbe it seems that the noun forms the basis for the derivation to the verb.
  16. genforhandling (renegotiation) can (in my opinion) be derived from at least two paths: gen+forhandling (re+negotiation) and genforhandl+ing (renegotiate+ion). The derived property in can contain both, but the combines property is not suitable for this case.
  17. Professortiltrædelsesforelæsning is another word where I am uncertain how to best decompose the word: professor+tiltrædelsesforelæsning or professortiltrædelse+s+forelæsning?
  18. What sort of word is politiker? A nomen agentis is (usually) derived from verbs, and with the analysis of the word into politik+er, then politiker is not a nomen agentis. But Den Store Danske notes that nomen agentis can be derived from other “nomens”, e.g., skuespiller (actor) and skuespil (acting or play). So is it ok to regard politiker as a nomen agentis?
  19. Some words for items that might appear as a collective is a singular concept in Wikidata lexeme and a collective in Wikidata, e.g., dansker (Dane) is danskere (Danes) in Wikidata’s Q-items. Connecting the entities via P5137, is a bit of a stretch. The same may be said to be an issue for animals and species.

Read also my paper Danish in Wikidata lexemes (Scholia) and perhaps also Validating Danish Wikidata lexemes (Scholia).

 

Photo in graph: Reading (7052753377).jpg, Moyan Breen, CC-BY 2.0.

SEMANTiCS 2019

Posted on Updated on

Nielsen2019Validating_poster

SEMANTiCS (Scholia) is a conference in the artificial intelligence/Semantic Web domain. It is a combination of an industry and a research-oriented conference. Among Danish SEMANTiCS 2019 participants there was (I believe) a 5-2 overweight of non-academics compared to academics. The conference has so far been held at different locations in Europe. Next year a spin-off conference is to be held in Austin. In 2020, there will also be a conference in Amsterdam.

SEMANTiCS 2019 (Scholia) was held in Karlsruhe, – the birthplace of much of semantic wiki technology. If my notes are correct, then there were 88 submissions to the R&I track, 20 long papers and 8 short papers were excepted with an acceptance rate of 27%. With only two-days for the conference there is a considerable less research than at ESWC (Scholia). The invoice amount for my registration was €637.

The first day was allocated to workshops and I attended the 1st International Conference on the European Industry on Language Technology (ELSE-IF 2019). It was the smallest conference I have ever attended. We could fit around a table, I guess around 10 people participated. Luc Meertens presented work that has been prepared for the European Commission regarding European language technology. A large report is supposed to be published soon. The European language technology industry is relatively small and I sense that there is the fear that American (and perhaps Chinese?) bigtech may take part of the cake. As far as I remember the SDL company may be the largest European company. There are several European language technology research project. I made a short improvised demonstration of Wikidata lexemes and Ordia in the workshop.

On SEMANTiCS 2019, Michel Dumontier keynoted on FAIR (findable, accessible, interoperable, reusable) data principles. Valentina Presutti spoke on the issue of common knowledge in the Semantic Web pointing to ConceptNet, Tom Mitchell’s NELL, Atomic, The human know-how dataset, FrameNet and Framester.

Under the title Language-related Linked (Open) Data for Knowledge Solutions, Artificial Intelligence and more, Felix Sasaki from Cornelsen and Christian Lieske fra SAP presented information on linguistic linked open data and Wikidata. They have recently written an online article Wikidata gets wordier, that mentions and screenshots Ordia.

To my knowledge the articles were not published before the event, neither for the main conference, nor the workshops. However, our (Katherine Thornton (Scholia), Jose Emilio Labra Goya (Scholia) and I) SEMANTiCS poster paper Validating Danish Wikidata lexemes (Scholia) has been available for some time now. The minute madness slides and the poster are now also available. I have added a few of the other papers to Wikidata so they show up in Scholia’s page for SEMANTiCS 2019.

Update 19 September 2019: Preproceedings were in fact available. Thanks to Harshvardhan J. Pandit for making me aware of the link.

On the road to joint embedding with Wikidata lexemes?

Posted on Updated on

road-to-joint-embedding

Is is possible to use Wikidata lexemes for joint embedding, i.e., combining word embedding and knowledge graph entity embedding?

You can create on-the-fly text examples for joint embedding with the Wikidata Query Service. This SPARQL will attempt to interpolate a knowledge graph entity identifier into a text using the short usage example text (P5831):

 SELECT * {
  ?lexeme dct:language ?language ;
          wikibase:lemma ?lemma ;
          ontolex:lexicalForm ?form ;
          p:P5831 [
            ps:P5831 ?text ;
            pq:P5830 ?form 
          ] .
  BIND(SUBSTR(STR(?form), 32) AS ?entity)

  ?form ontolex:representation ?word .
  BIND(REPLACE(?text, STR(?word), ?entity) AS ?interpolated_text)
}

The result is here.

The interpolations are not perfect: There is a problem with capitalization in the beginning of a sentence, and short words may be interpolated into the middle of longer words (I am not able to get a regular expression with word separator “\b” working). Alternatively the SPARQL query result may be downloaded and the interpolation performed in a language that supports advanced regular expression patterns.

The number of annotated usage examples in Wikidata across languages is ridiculously small compared to the corpora typically applied in successful word embedding.

Update:

You can also interpolate the sense identifier: Here is the Wikidata Query Service result.

Danish public domain authors publishing after the spelling reform of 1948

Posted on Updated on

One annoying feature with finding Danish language use examples for Wikidata lexemes is the Spelling reform of 1948 and the requirement of Creative Commons Zero license of Wikidata.

The Spelling reform of 1948 means that old public domain works in Danish, e.g., by Søren Kirkegaard and Hans Christian Andersen are with an old spelling which entails capital first letter for common nouns, the use of “aa” instead of the modern “å” and certain other spelling variations.

Works in Danish published after 1948 might have the new spelling (but verbatim reprints/republications of, e.g., Hans Christian Andersen’s works might still have the old spelling). Unfortunately the copyright law requires the author to dead for more than 70 years before his/her works fall into public domain and we can use it in Wikidata (It is unclear – to me at least – whether the use of short excerpts, e.g., a subsentence from a copyrighted work can be regarded as public domain). Given that we are now more than 70 years away from 1948 we might begin to be “lucky” to see works published after the spelling reform and where the author has died, e.g., in 1949. Such work will soon fall into public domain and we could use these in various context in Wikidata, particularly for the language use examples in Wikidata lexemes. Can we find such works?

My idea was to turn to Wikidata and formulate a SPARQL query against Wikidata Query Service for works published after 1948 and where the author has a death date. Here is one attempt:

SELECT ?work ?workLabel ?author ?authorLabel ?death_date WHERE {
  ?work wdt:P50 ?author .
  ?work wdt:P407 wd:Q9035 .
  ?work wdt:P577 ?publication_date .
  ?author wdt:P570 ?death_date .
  FILTER (YEAR(?publication_date) > 1948)
  SERVICE wikibase:label { bd:serviceParam wikibase:language   
    "[AUTO_LANGUAGE],da,en". }
}
ORDER BY ?death_date
LIMIT 100

The result is available here. Works of Steen Steensen Blicher, Søren Kierkegaard, H.C. Andersen, Meïr Aron Goldschmidt, Ludvig Mylius-Erichsen are in public domain and some of the works have been published after 1948. Some of Ludvig Mylius-Erichsen’s works are available on Wikisource, e.g., Julegæster fra havet. The version on Wikisource is with a modern Danish spelling. It has been used a bit for Wikidata lexemes, see the Ordia page for Julegæster fra havet: https://tools.wmflabs.org/ordia/reference/Q22084925.

Wikidata leksemer og Ordia

Posted on Updated on

ordia-danish-lexical-categoriesI 2018 fik Wikidata mulighed for at repræsentere leksemer (ordbogsopslag) og herunder deres former (dvs. bøjninger) og betydninger (på engelsk: senses). Wikidata-siderne for leksemer adskiller sig fra de almindelige emne-sider på Wikidata: Der er specielle felter til angivelse af sprog, leksikal kategori (ordklasse), grammatiske karakteristikker og for betydninger er der “gloss”-er. Idéen er at få Wikidata til at fungere som en struktueret og maskinlæsbar pendant til Wiktionary.

Da Wikidata, og dermed Wikidatas leksemer, er under Creative Commons Zero-licensen er det ikke umiddelbart nemt at finde gode leksikografiske resurser, og leksemerne er mere eller mindre indtastet manuelt. Der findes enkelte online værktøjer der letter indtastningen: Lucas Wekmeisters forms og Alicia Fagervings senses. Engelske leksemer er vel ikke overraskende dem der i øjeblikket er flest af. Fransk, svensk, nynorsk, polsk og tysk er også godt med. For dansk har jeg indtastet godt over 1.000 leksemer med tilhørende bøjninger og en del betydninger. Mange er linket til det danske ordnet der går under navnet DanNet. En del betydninger – særligt for navneordene – er linket til Wikidatas vanlige emner. Herfra kan man “gå rundt” i vidensgrafen og få hyponymer, hypernymer, synonymer og oversættelser.

Fyldigheden af Wikidatas leksemer både hvad angår antal leksemer og interlinkningen – er stadig noget svag og de forskellige ordbøger man kan skabe ud fra data (etymologisk ordbog, oversættelsesordbog, begrebsordbog, retstavningsordbog) er vel i øjeblikket noget sølle.

Parallel med indtastningen af leksemer har jeg udviklet og udvikler en webapplikation til at vise Wikidatas leksemer: Ordia. Den er tilgængelig fra Wikimedias computersky Toolforge. Da Ordia benytter Wikidata Query Service er det muligt at skabe sider på Ordia der samler information fra forskellige sider af Wikidata. I Ordia kan man for eksempel få en liste over alle bevægelsesverber eller navneord. Ordia har også en tekst-til-leksemer-funktion hvor man kan indtaste en tekst. Webapplikationen vil  udtrække ordene fra teksten, lave en forespørgsel mod Wikidata Query Service med ordene og vise matchede leksemformer og deres betydninger.

Der er stadig mange uklare elementer og åbne spørgsmål ved annoteringen af leksemerne. For eksempel, er den måde vi angiver at et verbum er et anbringelsesverbum brugbar? Skal den transitive og intransive udgave af verbet “hængte” være en eller to leksemer? Skal vi angive oversættelse ved hver enkelt betydning? Skal dansk s-genitiv angives i Wikidata? Kan vi med Wikidata specificere grammatik, således at det på sigt ville være muligt at skabe en grammatiktjekker? Hvad kan Wikidata leksemerne i det hele taget bruges til?

Ordia: Suggestion for a lightning talk at WikidataCon 2019

Posted on Updated on

Ordia is a Wikidata front-end running on the Wikimedia Toolforge https://tools.wmflabs.org/ordia/. Ordia displays information about the lexemes of Wikidata, including their forms and senses. It makes use of the Wikidata Query Service and can thus aggregate information from various different Wikidata pages. For instance, the language aspect shows statistics for the number of lexemes, forms and senses with respect to languages. Ordia also shows overviews over lexical categories, grammatical features, properties and the use of references. If a user input a text into a specific input field, Ordia can extract the individual words and query for the individual words. This talk will demonstrate the various uses of Ordia and briefly discuss the status of Wikidata lexemes.

Ideal number of attendees: 20

Take away: Attendees will know how to use Ordia and the limitation of Ordia and Wikidata lexemes.

Luftige spørgsmål til Wikimedia Strategi 2030

Posted on

Wikimedia forsøger at tænke langsigtet og lægge en strategi der sigter mod året 2030. Et udkast er tilgængelig fra https://meta.wikimedia.org/wiki/Strategy/Wikimedia_movement/2017/Direction

Her er nogle luftige spørgsmål der måske ville kunne få folk til at tænke over tingene:

  1. Hvorfor skal vi ha’ en strategi? Bør Wikimedia ikke blot udvikling sig organisk? Kan vi overhovedet forsige meget til 2030? Hvis vi ikke allerede kender vores strategi sidder vi så ikke allerede fast?
  2. Sidder vi fast i wiki-interfacet?
  3. Skal vi fortsætte med PHP MediaWiki interfacet som det primære software?
  4. Hvorfor er Wikiversity ikke blevet større, og slet ikke eksisterende på dansk? Er det fordi folk ikke gide lave Wikiversity? Er det fordi vi ikke ved hvad wikiversity er eller skal være? Er det fordi wiki-tekniske ikke fungerer i undervisningssammenhæng. Hvad skal vi ændre for at få det til at fungere?
  5. Hvorfor laver folk ikke flere video? Er det fordi at det er teknisk for besværlig? Er det for produktionsmæssigt for besværligt? Hvordan kunne Wikimedia hjælpe?
  6. Hvorfor er Stackoverflow det primære sted for faglige spørgsmål og svar? Burde det ikke have været Wikimedia der var det?
  7. Skal Wikimedia Foundation modtage penge fra firmaer så som Google? Vil det kunne skabe et afhængighedsforhold? Ifølge Peter Gøtzsches mening er patientforeninger påvirket i uheldig retning på grund af afhængighed til medicinalfirmaer. Kan Wikimedia-bevægelsen løbe ind i samme problem? Skaber det problemer med pengedonation, for eksempel i forbindelse med lobbyvirksomhede til EU’s ophavsretsdirektiv?
  8. Hvorfor kan OpenStreetMap kører med et mindre budget? Skyldes det langt mindre server load? Burde Wikimedia neddrosle og vælge en slags OpenStreetMap-model med hvor server værket bliver bedre distribueret til andre?
  9. “Knowledge equity” er et af to centrale begreber i Wikimedia Foundations strategi og noget svært at oversætte. Financial equity er hvad der på danske betegnes egenkapital. Et latinsk ord der nærmer sig findes i Den Store Danske, ellers er min nærmeste tanke det forældede udtryk “billighed”, – “ret og billighed” som det hedder i en dansk sang. Et sådant ord kan vi næppe bruge. Hvad kan vi på dansk forstå som “knowledge equity”?
  10. Kan Wikimedia komme i en situation som man har set Cochrane Collaboration hvor den professionaliserede del af organisationen kommer til at udmanøvrere græsrødderne? Hvad gør vi for at det ikke ske?
  11. Skal vi være stolt af at den danske Wikipedia stort set er opbygget gratis? Sidst jeg spurgte på den danske Wikipedias Landsbybrønd om Wikimedia Strategi blev det nævnt.
  12. Knowledge as a service følger en as-a-service-mønster man ser i datalogi. Her kan det hedder Platform-as-a-service e software-as-a-service. Hvad skal vi egentlig ligge i det? Jeg selv har skabt Scholia, et websted der viser videnskabelige data fra Wikidata via SPARQL-forespørgsler til Wikidata Query Service og Ordia, der gør det samme for leksikografiske data. Som sådan falder tanker om knowledge as a service fint i slag, – og jeg har da også forgæves forsøgt at erindre om det var mig der var med til at foreslå begrebet ved et internationalt Wikimedia-møde i 2017.
  13. Skal Wikimedia engagere sig i aktivisme, så som det sås til afstemningen om EU’s nye ophavsretsdirektiv? Har vi nogen succeshistorier på at det hjælper?
  14. Wikimedia Danmark har fået penge af Wikimedia Foundation til blandt andet et roll-up-banner. Det har været brugt i nogle få sammenhænge og vist været i tv. Er det sådan at Wikimedia Foundation skal bruge dets penge?
  15. Den visuelle editor synes at kunne hjælpe mange nye brugere, men er redigering af Wikipedia på en smartphone ikke meget besværlig? Kan man overhoved gøre noget ved det?
  16. Skal Wikimedia Foundation støtte forskere der bygger værktøjer eller undersøger fænomener på Wikimedia’s wikier?
  17. Normalt fungerer Wikipedia hurtigt, men hvis man kommer til et net der er langsomt oplever man at der kan være frustrerende at arbejde med, for eksempel Wikidata. Er det mon ikke frustrere at arbejde med wikier fra lande som ikke har hurtigt Internet? HVad kan der gøres ved det?
  18. Linux udvikles med en distribueret model, og sådan gør man med mange andre software systemer. Hvor er Wikipedia og andre Wikimedia wikier ikke distribuerede hvor fork og pull requests er nemt?
  19. Hvor mange af Wikimedia Foundations indsamlede midler skal anvendes på events, så som Wikimania?