tags: FME, ArcGIS Online, JSON Safe Software developed an ArcGIS Online Reader and Writer for use in FME. For basic use these are sufficient, but not very efficient. At this moment Attachments cannot be read with the Reader and dates cannot be written (correctly) with the Writer.
(2014-08-20: Note! The writer issue is solved in FME2014 SP3). This means another approach is necessary to get and to write Feature Services. But how? JSON is one answer! Esri offers a few different ways to edit your data in ArcGIS Online. All these methods are described in their REST API Reference. Hereunder I will describe two different methods for reading your data. In this blog I will not pay attention towards writing data, but that is well explained in this part of the REST API Reference and can be done with a HTTPUploader. Obtaining a token
The first important thing to know is that you need a token when you want to use data from ArcGIS Online. This token can be acquired very easy by using a HTTPUploader. Three attributes are required to do this: The most important part is the Target URL. The URL contains 5 variables. The first one defines the output, in this case JSON, the second and third the ArcGIS Online username and password. The fourth variable is the expiration time in seconds for the token and optional. The last variable is the website for which the token will be used. The Upload Method must be POST, this is the reason why only the HTTPUploader can be used for this operation. For my own Organization I have created a Custom Transformer (GetAgoToken) that obtains the token. I am working on getting the transformer published in the FME Store. Getting the data
For obtaining data from ArcGIS Online I compared two methods: 1. using the Query operation and 2. by creating a Replica database. In both cases it is important that you have knowledge about the data model of your Feature Service. 1. Using the Query operation
One of the REST operations that is available in ArcGIS Online is Query. This method is very easy to use to obtain all features. This can be done with a HTTPUploader, the same attributes are used as were used for the Token. The Target URL is shown herunder: http://services1.arcgis.com/$(Organization)/ArcGIS/rest/services/$(ServiceName)/FeatureServer/$(LayerID)/query?where=1=1&outFields=*&returnGeometry=true&returnIdsOnly=false&returnAttachments=true&f=pjson&token=@Value(token) In the URL three parameters are indicated: the Organization, the ServiceName and the LayerID. The Organization is coded by Esri, the ServiceName is the name of the FeatureService and the LayerID is the index of the Feature Class. For each Feature Class a request must be made. Next you can extract the information and the geometry by using JSONFragmenters and JSONFlatteners. My choice was to extract the “JSON[“features”]” and the “JSON[“fields”]". I used the last one to obtain the domain information and merged the domains to the features later on with a FeatureMerger.
figure 1. Example with a HTTPUploader A disadvantage of this method is that you still need to input the attributes manually in one of both JSON transformers or import them in a AttributeExposer. I prefer the last method and copy the attributes from the REST page of the service into a text file. The earlier mentioned Target URL can also be used in a FeatureReader. The advantage of this method is that all attributes and geometry directly become available. However, at this moment you first need to create a dump of the FeatureService locally, otherwise you cannot setup the FeatureReader. Like the attributes, the dump can easily be obtained via the REST page, by making a query and saving the result locally. After that you can set the ‘FeatureReader Dataset’ in the Navigator pane to an attribute. From FME2015 the attributes are directly available in the transformer wizard.
figure 2. Example using a FeatureReader and setting the Dataset from an attribute in the Navigator pane In both cases (HTTPUploader and FeatureReader) you only get the basic information, but you still don’t have the attachments. These can be obtained with a HTTPUploader and the two requests hereunder. Again the data can be extracted by using JSONFragmenters and JSONFlatteners: http://services1.arcgis.com/$(Organization)/ArcGIS/rest/services/$(ServiceNaam)/FeatureServer/$(LayerID)/@Value(OBJECTID)/attachments?f=pjson&token=@Value(token) http://services1.arcgis.com/$(Organization)/ArcGIS/rest/services/$(ServiceNaam)/FeatureServer/$(LayerID)/@Value(OBJECTID)/attachments/@Value(id)?f=@Value(contentType)&token=@Value(token) The first URL is used to check which features have attachments and get their attributes. The second URL is for obtaining the attachments themselves. 2. Using the Replica operation
The advantage of using the Replica operation is that you download a complete package, that contains all data including the attachments. It also allows to get a physical copy of your data in a file geodatabase (or JSON or HTML). The replica URL requires a bit more attributes:
http://services1.arcgis.com/$(Organization)/ArcGIS/rest/services/$(ServiceNaam)/FeatureServer/createReplica?replicaName=$(ReplicaName)&layers=$(LayerIDs)&returnAttachments=true&returnAttachmentsDataByUrl=false&async=false&syncModel=none&dataFormat=filegdb&f=pjson&token=@Value(token) The ReplicaName is a name you have to define yourself. If you choose another SyncModel than none, this name is important if you want to be able to sync changes in the original database otherwise it is not relevant. The name is not used for the database itself, it is just an administrative item of ArcGIS Online. The LayerIDs need to be separated by a comma without any spaces. The URL returns a file location that can be downloaded. Downloading the file can easily be done with a HTTPFetcher. It offers the possibility to directly save the file locally. The downloaded file can then be called with a FeatureReader (use the ArcObjects gdb reader and don’t forget to check the ‘Resolve domains’ in the reader parameters).
figure 3. Make a local Replica copy and read it with the FeatureReader A disadvantage of the Replica operation is that a physical database needs to be built and this might take time if lots of features are stored in the database and/or if many attachments or present. An example database with 300 features with attachments could take up to half an hour to be built. Within FME you will have to take into account that your connection might time-out. A solution for that problem can be found here. Conclusion
With FME there are many solutions for the same problem and one is not necessarily better than the other. In this case I prefer the Query operation above the Replica operation. If there are many domains in a dataset the Replica operation has some advantages, although they also can be obtained with the Query operation. But most of all if you want quick access to ArcGIS Online Feature Services do not use the ArcGIS Online reader, but one of the suggestions above. Looking at the time aspect, there seems to be a big difference between both methods. I must say that I only tested with a dataset with 17 features (and in total 18 attachments), but the Query operation took a bit more than a minute (for both methods), while the Replica operation almost takes two minutes for the same dataset. Taking into account that the Replica operation requires ArcGIS Online to build a database, it is not obvious that the Query operations seems to be faster. Tip!
Remember that dates in ArcGIS Online are stored in milliseconds from epoch (1-1-1970).