*Bounty: 50*

*Bounty: 50*

To give some background – the question is about measuring the **accuracy** **of** name disambiguation algo **results** (and not about the algo itself)

Let’s say we have three groups of entities, each corresponding to the same name (instead of real names they will be represented as numeric ids, just for simplicity). This is our ground truth:

```
+---------+------------+
| Groups | Duplicates |
+---------+------------+
| Group 1 | [1, 2, 3] |
| Group 2 | [4, 5, 6] |
| Group 3 | [7, 8] |
+---------+------------+
```

That means, that `[1, 2, 3]`

correspond to one name, `[4, 5, 6]`

for another one and so on.

Then, the results of our matching algo (not real data, just for example) look this way:

```
+---------+------------+
| Groups | Candidates |
+---------+------------+
| Group 1 | [1, 2] |
| Group 2 | [3, 5, 6] |
| Group 3 | [4, 7, 8] |
+---------+------------+
```

Knowing which candidates group corresponds to which group from he ground truth we can measure F1 easily:

But what if we don’t know which candidates group corresponds to which ground-truth group?

At first I tried “unpacking” groups, meaning I’ve created pairs of `id: other ids from the same group`

, for example, for ground truth `1: [2, 3]`

, `2: [1, 3]`

and so on, and measure the F1 for each such pair (between truth and prediction):

That gave me a result pretty far from the “real” one (the one where I knew the connections between groups)

Then I tried to create one-to-one pairs – `id: single id from the group`

, for example for ground truth `1: 2`

, `1: 3`

, `2: 3`

and so on.

Which gave me result far from desired as well (if I interpret it right, in this particular case accuracy equals precision, recall and f1)

And finally **the question is** – is there any way to measure how accurate the prediction is (without knowing which prediction corresponds to which group from ground truth) and at the same time to have that measure as close to the one where we knew the relations as possible?