Friday, June 15, 2018

Using FTDNA's API: Commands and Explanations

After hearing from all of you, I am now fairly certain that anyone is, in fact, allowed to access FTDNA's family finder API. And, as a bonus, I learned some new things about how APIs function in websites as well (special thanks for that goes to the anonymous commenter on the previous post). The question that remains is whether or not programmatically accessing it is against FTDNA's terms of use, and if anyone knows anything about that I'd be very happy to hear it, as I can think of a few things I want to try doing with the API and my best friend, python.

Anyways, this new post will go over the ways you can use the API and the different requests you can send to it. It's not all-inclusive and only includes the things I thought were important/most useful.

Please note, this post will include a lot of technical words without a lot of detailed explanation. If you just want to try it out for yourself simply click one of the links in the post to see the match data after signing into ftdna in your web browser.

How to Send GET Requests (In General)

There are two ways you can send Requests. The first is to just put the URL in your browser and the 2nd is to use Postman. For the most part, I use because making requests in it is faster and its easier to change parameters, however, I will use my browser in this tutorial because that will make it easier to collapse things/ limit the amount of censoring I have to do.

Match Lists

Basic Match List

The JSON Data for the Basic Match List

To return the information for your first 9 matches in JSON format (the same number as you would normally see on the screen) Use the following url:

Full Match List

You'll notice that the first line of JSON data returned by the last command has the total count of matches you have. You can use this number or any number larger than it in the url to print out that number of matches. In the url set page=1 (to start on your first page of matches) and set pageSize=numberofmatches. Here is an example url:

Fields in the JSON Raw Data

Now that you have printed out a full match list, we can inspect the raw JSON data returned for a single person. The person we will use is me through my Grandpa Tim's match list.
Here is my information as seen in the JSON data. Blue text are my explanations of each section:
   
      {
Unique number identifying the person who owns the kit you are logged into (my grandpa Tim). It is not the same as the persons actual kit number but is useful for the programmer.
            "resultId1": 3805399,
Unique number identifying the match (Me) 
            "resultId2": 3683806, 
Information regarding the Match's Name
            "name": "Renee Joanne Schmidt",
            "matchPersonName": "Renee Joanne Schmidt",
            "prefix": "",
            "firstName": "Renee",
            "middleName": "Joanne",
            "lastName": "Schmidt",
            "suffix": "",
The matches email
            "email": "RJS2018 at gmail dot com",
Whether or not the person has published a family tree on ftdna.
            "hasFamilyTree": true,
?
            "ffDataTypeId": null,
            "thirdParty": false,
Whether or not you have a note assigned to the person
            "note": true,
If you do have a note, the contents of that note
            "noteValue": "contacted",
Whether or not the person is female
            "female": true,
?
            "contactId": 0,
            "resultGUID": "00000000-0000-0000-0000-000000000000",

Base64 encoded string of the kit number. This decodes to a string of encrypted text representing the kit number, which is a good thing because it suggests ftdna takes strong security measures to protect your kit number from being available to the public (your user name).
            "kitEncrypted": "YmFzZTY0",
   
            "kitNum": null,
            "aboutMe": null,
            "paternalAncestorName": null,
            "maternalAncestorName": null,
Match Date
            "rbDate": "2017-05-28T23:56:28.1",
?
            "relationship": 7,
Family Finder number of shared segments on the x chromosome.
            "fF_XSharedSegments": 1,
List of the most likely relationships
            "relationshipRange": "Half Sister, Grandmother/ Granddaughter, Aunt, Niece",
Single most likely relationship
            "suggestedRelationship": "Aunt/ Niece",
?
            "ftrRelationshipGroupId": null,
            "ftrRelationshipName": "",
            "ffRelationshipGroupId": null,
Total shared centimorgans
            "totalCM": 1516.9967,
?
            "createDT": "0001-01-01T00:00:00",
The matches identified surnames, separated by pipes
            "userSurnames": "| Atkins| Allerton| Angel| Andersdotter| Andersson| Bishop| Belding| Belinska| Benes| Bangs| Bengtsdotter| Bray| Byers| Byers (Ohio)| Byers (Illinois)| Burnes| Burnes (Minnesota)| Burns| Brandenburg| Coffield| Coffield (Kansas)| Coffield (Virginia)| Coffield (Ohio)| Coffield (Missouri)| Coffield (Virginia)| Cooke| , (manually reduced in size for clarity's sake)
The longest centimorgan shared with the match
            "longestCentimorgans": 166.76224,
Haplogroup information
            "ydnaMarkers": "",
            "mtDNAMarkers": "",
            "yHaplo": "",
            "mtHaplo": "",
Whether or not they are an x match
            "isXMatch": true,
?
            "mySummary_FFBack": false,
            "matchSummary_FFBack": false,
The sides of the family ftdna has determined the match to match you on. In this case 3 means I match both sides of my grandpa's family.
            "bucketType": 3,
?
            "myKitRelease": false,
            "matchKitRelease": false,
            "relationshipId": null,
            "relationsGroupId": 1,
            "relationshipDistance": 999,
The row this person was on in the JSON output (in this case row 0)
            "rownum": 0,
A link to the person's tree on ftdna.
            "familyTreeUrl": "/my/family-tree/Share?k=k1PjcMet17nvkbjwCZ5wsg%3D%3D"
        }


ICW Data for Match List

If you look at the JSON data for an individual you will see there is a field titled kitEncrypted followed by a base64 encoded string. We can use this encrypted kit number to gather icw data for the person the kit belongs to and our kit by setting filterEncryptedId equal to the base64 string in our url like so (base64 string in the example is just the word base64 encoded in base64):

That will print out every other person that matches both you and the person belonging to the other kit.

Only Paternal or Only Maternal Matches

You can also return a list of only the matches ftdna has identified as paternal, maternal, or as both. This is defined by the setting selectedBucket. You can set this to one of 4 numbers, which each refer to the following:
0: Everyone
1: Paternal matches
2: Maternal matches
3: Matches to both Maternal and Paternal sides

Here is an example where it prints out only maternal matches:


Tree Information

Sample Request

In the section "Fields in the JSON Raw Data", the "kitEncrypted" field was mentioned. To get the family tree information of a match, you will need the base64 string assigned to this field. Note that I am going to go through the sections a bit more quickly now.

Here is an example request:
https://www.familytreedna.com/my/family-tree/getTreePeople?startId=&pedigreeView=false&generationCount=16&alwaysIncludeTreeOwner=true&treeKitNum=YmFzZTY0

The fields in the url above that you will most likely wish to change include:

  • startID (the id of the person to start the tree from)
  • generationCount (determines the number of generations back in the matches tree to include in the data
  • treeKitNum (here is where you put the 'kitEncrypted' string for the match whose tree you wish to see.

Fields in the JSON Raw Data

This time I will only go over the fields I feel are important and skip the others.
The unique number identifying a person within a tree; can be placed in the 'startID' field of the request to change the location the tree starts from. This is probably most useful for requesting data on person hints within the tree (not covered in this post but fairly easy to figure out).
        "id": 313566325,
The encrypted kit number of the person if they have been attached to the tree. As far as I can tell, it seems like this is nulled if you are not a match to the person as well.
        "personKitNum": "YmFzZTY0",

        "personKitNumDirty": false,
        "contactId": 319167610,
        "person": {
            "prefix": null,
            "firstName": "Renee",
            "middleName": "Joanne",
            "lastName": "Schmidt",
            "suffix": null,
            "gender": "F",
            "birthDay": 17,
            "birthMonth": 3,
            "birthYear": 2000,
            "birthPlace": null,
            "living": true,
            "deathDay": null,
            "deathMonth": null,
            "deathYear": null,
            "deathPlace": null,
            "email": "RJS2018@gmail.com",
            "location": null,
            "story": null,
The url to the matches profile picture
            "pictureUrl": "https://d3tije9h5o4l4c.cloudfront.net/social-photos/1880241?dpr=2&fit=crop&h=50&w=50&_=20170519074154",
            "newPictureId": null,
            "pictureCropX": null,
            "pictureCropY": null,
            "pictureCropW": null,
            "pictureCropH": null,
            "hasSurnamesChanged": null,
            "surnames": null
        },
        "paternalAncestor": null,
        "maternalAncestor": null,
        "imageFile": null,
        "notes": null,
The ids of the immediate family members of the person
        "motherId": 513393265,
        "fatherId": 513393264,
        "spouseId": null,
The generations difference from the start person
        "generation": 0,
        "dependsOnId": 313566325,
        "isTreeOwner": true,
Whether this person is a direct line family member of the start individual
        "isFamilyPedigree": true,
        "yHaploGroup": "-",
        "mtHaploGroup": null,
        "hasFamilyFinder": true,
        "isFamilyFinderMatch": false,
        "hasYTest": false,
        "isYTestMatch": false,
        "duplicateOfId": null,
        "yTestName": null,
        "hasMtTest": false,
        "isMtTestMatch": false,
        "mtMatchTest": null,
        "isMatch": false,
Whether or not this person is linked to a dna kit.
        "isLinked": true,
        "privacyLevel": 2,
        "isInspectable": false,

        "sendLinkNotification": false

Conclusions

This is by no means a comprehensive guide to everything you can do using the api, but it's a good start. You can look in the network requests in dev tools usually to find other api calls you can make.

7 comments:

  1. Hey Reneee,

    Have you noticed that FTDNA has changed things up? We use this API for our Match Manager program and sometime yesterday evening it stopped working for FTDNA. I've done some preliminary checking, and it looks like they switched from a GET, and all the parameters on the URL, to a POST and all the parameters in the body. There's also something going on with authentication and cookies that I haven't looked into yet. Are you seeing the same thing?

    You can ping me via the Rosetta Slack group if that's easier.

    Thanks,

    Dave

    ReplyDelete
  2. Actually it looks like just the switch from GET to POST. I wasn't setting things correctly on the XmlHttpRequest to set the cookies so that was causing me grief. I've now successfully run a test to this URL:
    https://www.familytreedna.com/my/family-finder-api/matches

    Using this as the post body:
    {"filter3rdParty":false,"filterEncryptedId":null,"filterId":0,"filterSince":null,"page":1,"pageSize":30,"searchAncestral":null,"searchName":null,"selectedBucket":0,"sortDirection":"desc","sortField":"relationshipPercentage()","trial":0}

    Dave

    ReplyDelete
    Replies
    1. I haven't actually checked up on it in a while, I'm glad you figured out a solution haha.

      Delete
  3. I was very happy to find your page. Like many, I have long wanted FTDNA to release an API, so your discovery is very welcomed news. You mentioned that your best pal is Python. Python and I are good friends, too. But, the FTDNA API is not. How have you logged in using Requests? My typical "session.post(login_url, data=payload)" method is failing me.

    ReplyDelete
    Replies
    1. Hello,
      I have seen your comment however it has been a while since I wrote this. I was planning on working on a new tool for ftdna, so once I get started on that if I find the solution to the problem I will let you know.

      Delete
    2. import requests
      cookies = {
      '_ga': 'GA1.2.1681402700.1545789431',
      'ai_user': 'HuWYR|2018-12-26T01:57:11.594Z',
      'ASP.NET_SessionId': 'nyxuo23isnypku4zwvxbrtzd',
      '_gcl_au': '1.1.1817641569.1547761586',
      '_gid': 'GA1.2.1634254588.1547761586',
      '_fbp': 'fb.1.1547761585775.963271947',
      '__RequestVerificationToken': '4gkD8_TzKLf5Y3wIxoxd_OWlXsSLEDqsm4QcyAAgEf63QSmiyBG8Oyf9YB_JbR8E8j5tSoDLvq7ePULEr27SuzOGBqXiQmsRqB9TnqqDLglKcqR9byIqRVvFlSD5aSO7o6qf_XCgM1a7mcye-xcEAQ2',
      'BNI_ServerId': '0000000000000000000000001b960c0a00005000',
      'ai_session': 'AKEnE|1547767829426|1547768364786.2',
      }

      headers = {
      'Origin': 'https://www.familytreedna.com',
      'Accept-Encoding': 'gzip, deflate, br',
      'Accept-Language': 'en-US,en;q=0.9',
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
      'Content-Type': 'application/json;charset=UTF-8',
      'Accept': 'application/json, text/plain, */*',
      'Referer': 'https://www.familytreedna.com/sign-in',
      'X-Requested-With': 'XMLHttpRequest',
      'Connection': 'keep-alive',
      'Request-Id': '|Nthjy.b03Q0',
      '__RequestVerificationToken': '7AkM4a6cWkrg8pE_o6yXvnrJ2k_WRMVvG__oA1AXYFmRPLMTc3VuhvlvtYPb_5Xf4x_5Y0mITA34oh4M275TSCEkD9p7iGoV7TwWh-847c7SIBq7brNH1yFQTcnJLoXNEnGmCkpDwOdsSYT6wsZ8xw2',
      'Request-Context': 'appId=cid-v1:8ec91be2-0790-49c2-aa80-f09f1e237de8',
      }

      data = '{"model":{"password":"Password,"kitNum":"KitNumber","rememberMe":false,"flow":""},"returnUrl":""}'

      response = requests.post('https://www.familytreedna.com/sign-in', headers=headers, cookies=cookies, data=data)

      This worked for me, you could try it as well.

      Delete
  4. Hmmm. Interesting. If nothing else, it shows how much I have to learn! I hope this inspires a new blog post from you. ;-) Thanks for sharing.

    ReplyDelete