When the geodatabase is upgraded to the latest release, as with ArcGIS 10, it’s not always possible for previous releases of ArcGIS to connect to and use those geodatabases.

You can create an earlier version of a file or personal geodatabase in ArcGIS 10 via geoprocessing tools. Specifically via the following “Data Management” tools which have an optional parameter allowing what version of the geodatabase you want to create.

   Create File GDB tool

> version 10, 9.3, or 9.2 file geodatabase

   Create Personal GDB tool

> version 10, 9.3, 9.2 or 9.1 release personal geodatabase

Stefanie L

This tip is to give a brief background to the two main ways to connect from your ArcGIS Desktop to your ArcSDE Geodatabase, more detail is available at;

http://webhelp.esri.com/arcgisserver/9.3/java/index.htm#geodatabases/arcsde-2034353163.htm

Two tier connection (also called direct connect)

Usually the preferred type of connection is via what is termed a two tier connection, where you connect directly from your computer to the database which will be on one of the corporate servers.  The term two tier refers to the first tier being the ArcGIS Desktop application and the second being the database being accessed.

A prerequisite for this is to have the appropriate database client software installed on your computer, for example if it is an Oracle database you are connecting to you will need an appropriate 32 bit Oracle Client installed.  Your IT department will be able to assist with installing and configuring this software if you do not already have it (you should already have the SQL Server Native Client software installed).  There may also be some additional database configuration to ensure you are using the correct account credentials to access this geodatabase (to just have access to the data you need) and that the database can support these additional connections.

The type of database you are connecting to will determine how to fill out the Add Spatial Database Connection properties, in the case below we are connecting to SQL Server on server lap-303750 using database authentication, although your administrators may prefer you using Operating system authentication in which case they can tailor data access just for you.

The following link provides more detail;

http://webhelp.esri.com/arcgisserver/9.3/java/index.htm#geodatabases/arcsde-2034353163.htm

Three tier connection

This is where there is another (application server) layer of ArcGIS software called ArcSDE on a server (in some case also on the database server) between your ArcGIS Desktop and the database.  The application server (ArcSDE service) is now the second tier, and the database the third tier.  This is usually slower as there is an additional service between you and the data you need.

Again the connection properties are different and the Service: name is either the name of the service or a port number like 5151.

Please note that for all connections using Database authentication that saving the username and password and sharing map documents (MXD files) which have this authentication information used within them is a potential security risk.

TADAAH! The Geodatabase Toolset (GDBT) for ArcCatalog.

This question has come up a few times from clients recently, and thought this article would be useful for other users wanting to make use of their DBMS spatial types.

At ArcGIS 9.3, a new geoprocessing tool (Migrate Storage), has been added which makes migrating your SDE Binary data to the new SQL Server spatial types such as Geography and Geometry spatial types quite simple.

This geoprocessing tool, allows you to choose the datasets that you would like converted, and the configuration keyword – such as SDO_GEOMETRY (for Oracle DBMS) or Geography (for SQL Server 2008 spatial types) – that you want your dataset/s converted to.

The Migrate Storage (Data Management) Tool can be accessed via ArcToolbox > Data Management Tools > Database toolset > Migrate Storage.

Tania T.

Up to and including v10.0, the ArcCatalog “Set Data Source” tool is only designed for updating MXD file based data sources such as Personal and File geodatabases.

ArcSDE data connections updated via the ArcCatalog “Set Data Source” tool are not supported. The tool may allow you to update or repair MXD data references from ArcSDE geodatabases but is not the recommended method. 

The recommended methods are:

# v10.0 #
Use ArcGIS v10.0 map scripting with Python via the new arcpy.mapping module.

Online ArcGIS Desktop 10.0 Help > Updating and fixing data sources with arcpy.mapping

# PRE v10.0 #
If the workflow exists where new server and database are both setup, organisation that have multitudes of dedicated MXD’s pointing to an old server can re-direct the connections. Testing has shown a valid dummy database connection set up for the broken links will cause the MXD to time-out and open with the option to repair data sources nearly as quickly as default behavior.
OR
via the following unsupported ArcScripts:
- Change ArcSDE Datasources
- Automate datasource change in MXD’s and MXT’s
- ArcMAP MXD Redirect Data Sources
- Generic Set Data Source for any MxDocument

NOTE: I have not personally tested the ArcScripts. Esri does not support ArcScripts.
ArcScripts are intended for the free exchange of scripts and tools related to Esri software products. For any related queries on these scripts please contact the script author.

Further details in Esri Inc’s knowledge base technical article below:
27425 HowTo: Use the ‘Set Data Source’ tool for map documents in ArcCatalog

Stefanie L

This blog aims to highlight the fact that ESRI Australia staff are actively contributing to the content of the main ESRI Inc (US) web site, as well as our own. In fact ESRI Inc is pushing for all local distributors, across the globe to ramp up the amount of  ’ESRI Global Content’ they submit.

Well, here is a link to our first contribution (of many…) to get the official sign off: http://support.esri.com/en/knowledgebase/techarticles/detail/39660.

This document is a ‘How To’ which adds to the content found in ESRI’s ‘Knowledge Base – Technical Articles’ web page: (http://support.esri.com/en/knowledgebase/techarticles). In terms of subject matter, it covers the considerations and steps required for publishing Mosaic Datasets to ArcGIS Server. In light of the fact that ArcGIS 10 is the last release of the stand-alone ArcGIS Image Server product, this is the method we will use when creating image services from now on, and as such the document should hopefully be useful to many. To give a bit of background to this, the Image Service Definition (.ISDef) used by the standalone version of Image Sever  has been replaced  by the Mosaic Dataset. This can be published as an image service using the new ArcGIS Server Image extension and removes the need for a separate image server.

Please make sure you read the article and keep an eye out for more Australian content on the main ESRI website (there will be an Australian flag!).

John H

Compressing the ArcSDE geodatabase pushes records from the Adds & Deletes tables to the base table.
Over time, an actively edited geodatabase will accumulate unreferenced rows in its delta tables and a deep state tree, both of which can negatively impact performance. Occasionally, depending on the volume of edits, the ArcSDE administrator must compress the geodatabase to remove the unreferenced rows and reduce the depth of the state tree. Compressing never removes data that can be accessed through any version’s lineage and will not harm your geodatabase; it exclusively cleans up unused data.

If you are finding that there are still records in the A & D tables after running the SDE compress, please try the following steps before performing the compress operation again:
1) Reconcile and Post all outstanding changes in child versions to Default.
2) Sometimes it helps to delete the child versions, and have it re-created after a full compress is achieved.
3) Synchronize all Replicas. Remove (unregister) any replicas that are no longer being used.
4) Ensure no users are connected.
• Stop the SOC and SOM services (which in turn should stop any map services referencing the ArcSDE data).
or;
• Stop the map services via ArcCatalog or ArcGIS Server Manager.
• Check if there are any users still connected by running the sdemon -o info -I users
• Run the sdemon -o kill -t all command to disconnect all connections.

sdemon -o kill -t <{ all | pid }> [-p <ArcSDE_admin_password>] [-N]
{[-i <service> | <port#> | <direct_connect_info>]
{[-s <server_name>] | [-H <sde_directory>]}
[-u <user_name>] [-p <user_password>] [-D <database_name>]

5) Try Compressing again.
• The compress command is best performed at night when there are fewer chances of other users being connected to the ArcSDE.
• Run the sdeversion -o compress command using the ArcSDE administrator user

sdeversion -o compress [-N] -u <ArcSDE_Admin_user_name> [-p <ArcSDE_Admin_user_password>] [-q] [-i <service> | <port#> | <direct connection>] [-s <server_name>] [-D <database_name>]

• Check that the states has been trimmed or check the A and D tables to ensure that there are no entries in there, which should tick all the boxes that the full compress was successful.

Tania T.

The long awaited GUI for managing ArcSDE administrative tasks such as versioning, connections and locks has arrived at 10.1!

The Geodatabase Administration Dialog can be accessed from ArcCatalog & ArcMap by right clicking the geodatabase connection.

Version Manager

The redesigned Version Manager has been moved into the Geodatabase Administration dialog box and includes the following new features:

• A tree view that lets you see how versions are related
• The ability to filter for a specific transactional or historical version, versions that could prevent the DEFAULT version from compressing to state 0, or versions owned by a specific user
• The ability to see all the properties of a selected version without having to open a separate dialog box
• A list recommending the order in which versions should be reconciled. Geodatabase administrators (sde/dbo) have access to the Reconcile Order subtab on the Geodatabase Administration dialog box. The subtab will only be available if you are signed in as the administrator.

More information can be found via the following web help link:

Version Properties:
http://resources.arcgis.com/en/help/main/10.1/index.html#//003n000000wm000000

Connections

You can use the Connections tab on the Geodatabase Administration dialog box to view all users connected to ArcSDE geodatabases. The information provided helps you monitor which users and client machines are connecting to your geodatabase and how long each session has been open.

You can disconnect user sessions from this dialog box by right-clicking a session and clicking Disconnect User.

In order to view ALL connections, you will need to be connected as the geodatabase administrator such as the sde/dbo user. The “Connections” tab is only visible, once a connection as the geodatabase administrator has been made. This will allow you to view both Direct and Application server connections.

More information can be found via the following web help link:
Viewing connected users in ArcGIS for Desktop
http://resources.arcgis.com/en/help/main/10.1/index.html#/Viewing_connected_user_sessions/019000000003000000/

Locks

The Locks tab allows you to view what type of locks are being held on data in an enterprise geodatabase and who is holding the lock. Through this dialog box, you can see active locks on datasets and versions and which user has acquired the lock.

In order to view ALL locks, you will need to be connected as the geodatabase administrator such as the sde/dbo user.

More information can be found via the following web help link:
Managing locks in the geodatabase
http://resources.arcgis.com/en/help/main/10.1/index.html#/Managing_locks_in_the_geodatabase/01900000000n000000/

Tania T.

Spatial views are database views that contain a single spatial column. They are useful in organising your spatial data with other attribute data in a predefined way, in the same database or another database. They are also READ-ONLY, hence spatial views can not be edited via ArcMap. If you are currently still at ArcSDE 10.0 and prior versions, the trick to get this working is to follow these steps:

This blog addresses how to create a spatial view using SQL Server.

In this example, we have some Parcel data and some Owner information in a table, and would like to join the OWNER_NAME from the Owners table to the Parcel spatial data on PARCEL_ID, which exists in both tables.

Sample data:

1. #First create a simple view

For example:
sdetable -o create_view -T VW_PARCEL_OWNERS -t PARCELS -c objectid,shape -i sde:sqlserver:server_name -D testdb -u test -p xxxx

2. #Alter or Modify the view in SQL Server Management Studio

It is much easier to alter the view in SQL Server using SQL tools, than specifying all the tables, fields and query definition (where clause) in the “sdetable –o create_view” command.

For example:

#Query Design:
ALTER VIEW SDE. VW_PARCEL_OWNERS AS
SELECT TEST.PARCELS.OBJECTID, TEST.PARCELS.Shape,
TEST.PARCELS.PARCEL_ID, TEST.OWNERS.OWNER_NAME
FROM TEST.PARCELS LEFT INNER JOIN
TEST.OWNERS ON TEST.PARCELS.PARCEL_ID = TEST.OWNERS.PARCEL_ID

Or,

# View Design

Add the OWNERS table and create the join; or, modify the Select statement.

3. #Grant TEST permission to external table in SQL Server (Optional).

If the OWNERS table is owned by DBO or another database user account (other than TEST, in this exercise), or in another database, the following permissions would need to be granted.

For example:
GRANT ‘Select’ permission for TEST to the non-spatial table.
- In SQL Server Management Studio, Right-click OWNERS table, …Properties, …Permissions, …add TEST user, …grant ‘Select’. Click OK.

To allow other users to access the view, grant privileges on the view to other users.

4. #Result

By default, the join type for all views created by the sdetable command is INNER JOIN.

If you needed to include all Parcel data even if no matches were found in the OWNERS table, change the View so that it uses an outer join, and Save the View. In this case, we have changed it to a LEFT OUTER JOIN.

5. #LEFT OUTER JOIN Result

Tania T.

At ArcGIS 10.1 & 10.2, the SQL Server Native client is not automatically installed with ArcGIS Desktop (as per previous ArcGIS Desktop versions) – it must be explicitly installed.

The SQL Server Native client can be downloaded from the Customer Care Portal.

Customer Care Portal: https://customers.esri.com

Login. Under My Software > Click Software Download

For 10.1:
Under DBMS Support Files > Click DBMS Support Files for ArcGIS 10.1

Under Microsoft SQL Server 2008 R2 Native Client > Choose either the 32-bit or 64-bit version > Click Download button.

For 10.2:
Under DBMS Support Files > Click DBMS Support Files for ArcGIS 10.2

Under Microsoft SQL Server 2012 SP1 Native Client > Choose either the 32-bit or 64-bit version > Click Download button.

More Information:
http://wiki.gis.com/wiki/index.php/ArcGIS_10_vs._ArcGIS_10.1#Client_libraries_for_Geodatabase_Connection
Under Geodatabases > “Client libraries for Geodatabase Connection”

“The installation of ArcGIS 10.1 doesn’t include client libraries for the various relational database management systems (DBMS). These have been provided on the Esri Customer Care Portal for user downloads to set up connections to the intended DBMS.”

It is also further reiterated via the following ArcGIS web help:

Database clients
http://resources.arcgis.com/en/help/main/10.1/index.html#//019v00000009000000

“…You can obtain DBMS client installations from the DBMS vendor or download them from the Esri Customer Care portal. “

Tania T.

I’m sure you’ve seen this option when choosing to “Register As Versioned”, and if you have wondered…”What does this option do?”…let me explain.

Checking this option, will still create the (A)dds and (D)eletes tables to track versioned edit operations performed on the data; however, edit operations that are saved to the Default version, or merged from other child versions, are immediately saved to the business table (rather than stored in the A and D tables).

Thus, the advantage is that edits are stored in the A and D tables while editing; giving you the undo and redo ability while editing in ArcMap – but then pushing the edits into the business table, upon Save.

This is great for 3rd party applications that query the business table and requires edits through ArcGIS to be seen immediately.

However there are some limitations:
• Available for simple features only – those that do not participate in a topology, network dataset, or geometric network.
• Cannot specify this option if the data you are planning to register as versioned, will be participating in geodatabase replication; or/and the data has geodatabase archiving enabled.

If you decide to make use of this option:
For 10.0:
• In the Catalog tree, right-click the feature dataset, stand-alone feature class, or table.
• Click Register As Versioned to open the ‘Register As Versioned’ dialog box.
• Check ‘Register the selected objects with the option to move edits to base.’

For 10.1:
• In the Catalog tree, right-click the feature dataset, stand-alone feature class, or table.
• Click Manage, and then click Register As Versioned to open the ‘Register As Versioned’ dialog box.
• Check ‘Register the selected objects with the option to move edits to base.’

Tania T.

Unfortunately, the ArcSDE geodatabase cannot be renamed in SQL Server.
This also applies to moving or copying an ArcSDE geodatabase from one server to another using SQL Tools such as Detach and Attach, Backup and Restore, or the Copy Database wizard.

No matter which of the methods you use to move your SQL Server database, you cannot rename the database. When you are restoring a database, for example, you are given the opportunity to restore it with a different name. Don’t do this with a geodatabase; you won’t be able to connect to it.

All object names in the geodatabase system tables are fully qualified with the database name. In addition, many stored procedures use a three-part naming syntax in their code, which follows the format <database>.<owner>.<object>. If the database name changes, you will not be able to execute these procedures.

This is further reiterated in the ArcGIS Resource documentation referenced from: http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#//002q00000087000000

Tania T.

With the release of 10.2 and plans to deprecate the ArcSDE command line tools, there has been lots of talk about how all these tools will be replaced.

The following technical blog and forum addresses some of the questions you may have regarding the deprecation of the SDE command line tools:

Do This, Not That! – Alternatives to using SDE command line tools
http://blogs.esri.com/esri/supportcenter/2013/10/04/do-this-not-that-alternatives-to-using-sde-command-line-tools/

Alternatives to using SDE command line tools – Blog Discussion
http://forums.arcgis.com/threads/95154-Alternatives-to-using-SDE-command-line-tools-Blog-Discussion

As you can see, even though there are plenty of alternatives in place already, Esri is still currently looking into users requests to have more SDE command line functionality replaced via GP tools or the GUI.

Tania

This is a recurrent question in my geodatabase seminars.

“ArcSDE” is a concept that has always created some confusion, and part of the reason is, I think, due to how ArcSDE has evolved overtime.

I have seen that some users tend to associate the concept ArcSDE with the “application service connections” which has been deprecated in 10.3.  Sometimes, I see people mixing things up and ending up asking questions like, “Has Esri got rid of ArcSDE?”

It’s also common to see users asking in forums why they should use ArcSDE if there are many databases that support spatial data, multi-user editing, replication,  and even spatial functions, as one of colleagues here in the Melbourne office found the other day. The confusion comes from the terminology used:

• Do I get the same spatial capabilities using native database spatial data versus ArcSDE enhanced spatial databases – even when both use the same spatial data types – for example,  do I get the same capabilities using a MS SQLServer database using the Geometry spatial data type and a MS SQLServer ArcSDE enhanced database using Geometry spatial data type?
• Is it the same multiuser editing found in SQL Server or any other database and the multiuser editing available with ArcGIS?
• Is it the same “replication” functionality the one found in a database and the one we have with the Esri technology?

A short answer to all these questions is: – No, it is not the same.

ArcSDE enhances the capabilities of the database (multi-user editing, spatial capabilities, etc.) to allow the implementation of complex GIS workflows that otherwise could not be implemented. In addition, the ArcSDE technology together with other components and features of ArcGIS platform, allow complex GIS workflows to work in an integrated way. Let’s see this with an example:

An organization that requires to implement multiple editing workflows (web editors, field data collectors, local editors, remote editors, etc.) and to consolidate the edits in a central enterprise geodatabase – or single point of truth – can do it easily, and in an integrated way using the ArcGIS platform. What a user captures in his/her device (no matter when, where they are or the device they use) is sent to the central geodatabase in a automatic and transparent way. There is no need of doing any manual intermediate step, changes go smoothly from the client to the server,   and this is so for all the editing workflows.

This “continuity” allows that all the GIS workflows of the organization can be implemented in a coordinated and integrated way and this fact – directly or indirectly – is facilitated by the ArcSDE technology.

Therefore, when we talk about “a single point of truth”, enterprise geodatabases, multiuser editing, replication, web editing, domains and subtypes, mosaics, networks, etc. That all implies that the use of ArcSDE. When we talk about the implementation of different enterprise deployment strategies using the ArcGIS technology (centralised, distributed operations) we are considering the use of the ArcSDE technology.

So, why is ArcSDE important…?

Firstly because it’s in the core of the ArcGIS technology and as such, it’s a key component of the ArcGIS platform. It allows the implementation of complex GIS workflows, and the deployment of the ArcGIS platform to support almost any business case, and second because it can be the key driving factor, that directly or indirectly, helps an organization to decide to go from a basic GIS implementation to a complete enterprise GIS solution.

Let me finish answering the question that gives the title to this post, Is ArcSDE dead? the answer is a rotund No, it is not. We are just changing our vocabulary. From version 10.3 onwards, ArcSDE is embed into the term “multi user geodatabase”.

Regards, Walter Simonazzi

Here’s one way forward utilizing ArcGIS for Desktop and Python….

Working at Esri Australia in the Support and Training realm, we listen to numerous client’s requests, concerns and workflow issues. To keep abreast of how our technology is heading, one source I frequent is the ArcUser magazine published with tutorials, tips and articles on best practise. In the Winter 2015 edition of ArcUser, there was an article about “Creating a Python Tool That Summarises ArcMap Layer Properties”. By no means is this article to take credit for this innovative post, but more to suggest how each and every ArcUser can adopt and enhance such functionality.

From this, I quickly knocked up some python script that made good use of arcpy.mapping classes and functions to return a useful summary of layer properties within each MXD that is entered as input. For each layer, the following list of properties [mxdpath, layer.longName, layer.description, layer.dataSource, mxd.author, mxd.dateSaved] was then written to a CSV output (see Figure 1 for exact python syntax guidance).

Figure 1. Excerpt of the functional logic returning layer properties within the python source file “MXD_layer_properties.py”

Once I tested and returned a list of layer properties, I wanted to share this functionality with all my ArcGIS users, so I created a script tool from this (a great reference to do this is here: http://desktop.arcgis.com/en/desktop/latest/analyze/creating-tools/a-quick-tour-of-creating-tools-in-python.htm), where Figure 2 shows the result of this script tool, a familiar geoprocessing tool dialog box.

Figure 2. Dialog box of the script tool “MXD Layer Properties saved to (3) optional CSV outputs”

If you are wondering why the script tool (as seen in Figure 2) has two optional inputs, MXD Directory Path OR MXD Single Path – it’s always a good idea to create your workflows considering how your users might want to use your script tool – this conditional functionality (where the code is shown in Figure 3) allows the user to return layer properties within many or simply an individual map document.

Figure 3. Excerpt of the conditional logic for input workspace within the python source file “MXD_layer_properties.py”

With the majority of the hard work complete (that being a tested python script and script tool created), I wanted to further enhance this to include two more separate CSV optional outputs returning layer symbology (e.g. [layer.showLabels, layer.symbologyType, layer.symbology.valueField] and layer properties of my service layers (e.g. layer.serviceProperties). The serviceProperties of any layer provides access to connection information for ArcSDE and/or web service layers. The returned results are dictionary key-value pairs, providing very useful information such as service type, server name, database, username and version for any ArcSDE layer; and for web services, important dictionary keys can be returned such as the service type, URL and service name.

My interpretation of the ‘why’ behind returning such large scale information is in gaining valuable insight as to each ArcGIS user’s interpretation and spatial analysis of each map document layer that is conveyed to their intended audience.

It only takes a matter of seconds to run the script tool “MXD Layer Properties saved to (3) optional CSV outputs” (provide below), where Figure 4 shows it is possible to view the completed layer property output in either ArcMap or another text editor, such as Notepad++.

Figure 4. CSV Output of the script tool “MXD Layer Properties saved to (3) optional CSV outputs”.

For those interested, I can email you a zip file comprising the script tool, source python file and three CSV optional inputs – my correspondence email is csherwin@esriaustralia.com.au. If you wanted to adopt and enhance such a script for your own value added workflows, why not take a look at the alphabetical list of the many arcpy.mapping classes and functions:

Alphabetical list of arcpy.mapping classes:

http://desktop.arcgis.com/en/desktop/latest/analyze/arcpy-mapping/alphabeticallistofclasses.htm

Alphabetical list of arcpy.mapping functions:

http://desktop.arcgis.com/en/desktop/latest/analyze/arcpy-mapping/alphabeticallistoffunctions.htm

Happy coding and enjoy utilising arcpy classes and functions!

Thanks for your time in reading this article,

Chris Sherwin

The following are useful links for your consideration:

What is the ArcUser? http://www.esri.com/esri-news/arcuser/what-is

ArcUser Winter 2015: http://www.esri.com/esri-news/arcuser/winter-2015/create-a-python-tool-that-summarizes-arcmap-layer-properties

Python courses available for you to consider: https://esriaustraliatechblog.wordpress.com/2014/08/01/wondering-how-to-use-python-with-arcgis-esri-australia-have-courses-that-can-help/

The term ArcSDE/SDE is being gradually replaced with the term “Multiuser Geodatabase” and this is probably creating some confusion among the users of the ArcGIS platform. In order to understand the difference between ArcSDE and multiuser geodatabase, let’s start clarifying what is and what is not ArcSDE.

What is not ArcSDE:

ArcSDE is not a product. It’s a technology. In the same way ArcGIS is not a product, is a platform. Only prior to ArcGIS 9.2, ArcSDE was a standalone software product. At the ArcGIS 9.2 release, ArcSDE was integrated into both ArcGIS for Desktop and ArcGIS for Server.

ArcSDE is not only the application sever connection, a piece of software that packs the giomgr.exe and a gsrv.exe processes. This is a common mistake that I have mentioned in my previous post but from a more business perspective. I’ll try to do the same exercise now but from a technical perspective.

So, what is ArcSDE  – also called “the database enhanced with the ArcSDE technology” or simply the “ArcSDE geodatabase” or the “Multiuser geodatabase”?

Let’s answer this question step by step.

First of all, ArcSDE is a hybrid technology built up from different components. These components are implemented in two places, the client (ArcGIS for Desktop/Pro or ArcGIS Server, mobile, etc.) and in the relational database management system (any of the supported ones).

The components that make the ArcSDE technology are those shown in the slide number 3 and 4 below.

Note that, the above are the slides of the official geodatabase training course version 10.2, taught all around the world therefore, at least until version 10.2, we have been using, explaining and discussing what ArcSDE is.

The slides below are from the same course at version 10.3. Note that the content is the same although the term ArcSDE has been replaced by “Multiuser geodatabase” (except that the translator component is only in the client side reflecting the deprecation of the application server connection).

What are the components of the ArcSDE/Multiuser geodatabase technology? – Note that the lack of any of the components described below would mean that the multiuser geodatabase wouldn’t work.

• The client (ArcGIS for Desktop (32 bits) , ArcGIS Pro and ArcGIS Server (64bits), etc.)
• The repository of ArcSDE/multiuser geodatabase, the geodatabase system tables, stored procedures and functions.
• The management tools:
  • Geoprocessing tools
  • Database tools – for example, the functionality to backup and restore is considered part of the ArcSDE technology.
  • And until version 10.2.x, the command line tools. Deprecated in version 10.3.

• And finally, the translator, this is the component that translates every map interaction (clicks on the map performed by the user) into SQL statements – Since the database doesn’t understand “clicks on a map”, it only understands the SQL language, we need something that acts as an “interpreter” between the user and the database.

Notice that in the slide above (version 10.2) the translator component is floating in the middle of the client and the server. There is a reason why this is so… the reason is related with how the client connects to the multiuser geodatabase. Up to version 10.2, there were two different connection types, application server connection and direct connections – first released with version 9.2 -. Let’s review some of the particularities of each connection type:

• Direct connections (two tier connection): If the client connects using direct connections, the translator component is located in the client side and requires to have the database client installed in the machine where the ArcGIS client is installed -. This is the default connection method.

• Application Service connections (3-tiers connection). The application server connection has been the traditional way to connect to an enterprise geodatabase. It requires a service (commonly called “sde” or “arcsde”) which is normally installed and configured in the same machine where the database server is installed.

The sde service packs two processes– the giomgr and the gsvr – the giomgr.exe is in charge of listening for incoming connections (clients creating a connection to the enterprise geodatabase using the sde service), when the connection is accepted, the giomgr fires up the “gsvr.exe”, one per client connection. Each gsvr.exe will then translates the clicks on the map performed by the user into SQL statements that are then passed to the database, the database interprets the SQL, generates the answer which is finally sent to the client through the gsvr.exe process, as it is shown below.

With this connection type all the translation load is placed on the server side.

As I mentioned before, the application server connection method has been deprecated in version 10.3 and with it, the command line tools.

I’m sure there are many reasons why the application server connection has been deprecated. From my point of view there is one reason that deserves to be discussed more in detail. Performance.

When the application server connection was released the processing capacity of the servers was much higher than the processing capacity of the workstations. The application server connection was a good approach to put all the translator processing load into the server machine and so to use the server processing capacity to perform the translation.

Nowadays however, there are reasons why we wouldn’t want to use application server connections:

• Today, we can use client’s machines processing capacity to perform the translation and in this way, to share the translation load among all the clients connecting to the multiuser geodatabase. We just need to install the database client and to connect to the multiuser geodatabase using direct connections. In this way, the translation will happen in the client machine, leaving the server to use the processing capacity for other purposes.

• Application server connections also create single points of failure, if the sde service fails, no one can connect to the geodatabase whereas with the direct connection this risk doesn’t exist.

• Application server connections put much more load on the server if we compare it with direct connections.

With regards to the last point, the capacity planning tool is an excellent tool to model each connection type and to see how much server processing capacity is required. Below I have done a quick exercise that shows the amount of server processing power required to perform the same workflow using application server connections versus direct connections.

I have modelled a simple workflow, ArcGIS for Desktop 10.2 connecting to a multiuser geodatabase using direct connection [1] and application server connections [2]. Both workflows generate 50.000 transactions per hour (140 concurrent users approx. with a productivity of 6 displays per minutes as average per user (a standard user).

[1] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using direct connection (DC).

[2] Workflow: ArcGIS Desktop connecting to a multiuser geodatabase using application server connection (ASC).

Notice that using direct connection the workflow requires 14,7% of the server processing capacity whereas using application service connections it requires 29,3% of the server processing capacity, almost double. This is another reason  to reconsider the use of application server connections, if you are still using them. 

I hope I was able to bring some light into this topic. If you are interested in expanding your understanding about enterprise geodatabases, I highly recommend to attend the following training courses (in particular the second one):

Deploying and Maintaining a Multiuser Geodatabase (10.3)

Implementing Versioned Workflows in a Multiuser Geodatabase (10.3)

Regards, Walter Simonazzi

Question

How do you turn on verbose logging in ArcGIS for Desktop when doing a direct connection to an enterprise geodatabase?

Answer

Method 1

You should add

        ;SDEVERBOSE=TRUE

at the end of the Instance field in the properties of your database connection. So if your Instance field contains:

        your_datasource

then to turn on SDEVERBOSE logging you would enter into the Instance field:

        your_datasource;SDEVERBOSE=TRUE

Verbose information will be written to the direct connect log in the %TEMP% directory. To locate this file open Windows Explorer and type in the address bar

        %TEMP%

Press Enter.

You will be taken to the TEMP directory where the direct connect log can be found: This file has the name sdedc_{database}.log.

For example, for SQL Server the file is called sdedc_SQL Server.log, for Oracle the file is called sdedc_Oracle.log, and for PostgreSQL the file is called sdedc_PostgreSQL.log.

To turn off SDEVERBOSE logging, it is recommended to delete the connection where SDEVERBOSE was set to TRUE and close ArcGIS for Desktop and open it again. Otherwise any other database connection after using the SDEVERBOSE connection will also have SDEVERBOSE set to TRUE.

Method 2

If you are making a direct connection to the geodatabase, you can create an etc directory under the client installation location. For example, for ArcGIS for Desktop this may be

        C:\Program Files (x86)\ArcGIS\Desktop 10.x\etc

In this directory create a file called dbinit.sde to be read when the client connects directly to the geodatabase. In this file add the line:

        set SDEVERBOSE=TRUE

To turn off SDEVERBOSE update the line in dbinit.sde to be:

        set SDEVERBOSE=FALSE

or comment out the line:

        # set SDEVERBOSE=TRUE

and disconnect and reconnect to the database connection.

Greg B.

An ArcGIS geodatabase is a collection of geographic datasets of various types held in a common file system folder, or a multiuser relational database management system such as IBM Db2, Microsoft SQL Server, Oracle, PostgreSQL, or SAP HANA.

Several cloud providers, such as Microsoft and Amazon, offer managed database services that are also supported by ArcGIS Enterprise, and you can create a geodatabase in these database service offerings as well.

Refer to the following resource for further information about the requirements for using ArcGIS Enterprise and ArcGIS Pro with databases in the cloud.

Requirements for using ArcGIS 11.0 and ArcGIS Pro 3.0 with databases in the cloud – https://enterprise.arcgis.com/en/system-requirements/latest/windows/databases-in-the-cloud.htm

There are two types of databases in the cloud: databases that are installed on virtual machines in the cloud, and databases that are available as database-as-a-service offerings. For database-as-a-service types, you do not have access to the machine that serves up the database and you do not have to perform maintenance tasks such as applying security patches on the database or operating system.

When talking about Microsoft Azure’s database-as-a-service offering, we use the term “Azure SQL Database”.

Azure SQL Database – https://azure.microsoft.com/en-au/products/azure-sql/database/#overview

Although Esri’s ArcGIS Enterprise Cloud Builder for Microsoft Azure (https://enterprise.arcgis.com/en/server/latest/cloud/azure/what-is-arcgis-server-cloud-builder-for-microsoft-azure.htm) can be used to add databases to an ArcGIS Server site on Microsoft Azure, in some circumstances clients may wish to manually create Enterprise Geodatabase in an Azure SQL Database with a geodatabase administrator named “sde”.

The geodatabase administrator in SQL Server – https://pro.arcgis.com/en/pro-app/latest/help/data/geodatabases/manage-sql-server/geodatabase-administrator-sqlserver.htm

A comparison of geodatabase owners in SQL Server – https://pro.arcgis.com/en/pro-app/latest/help/data/geodatabases/manage-sql-server/comparison-geodatabase-owners-sqlserver.htm

This blog article can be used as a reference for the workflows required to create an Azure SQL Database with an ”sde-owned” Enterprise Geodatabase.

Create an Azure SQL Database from Azure Portal.

1. Navigate to the relevant Resource Group.
2. Click on Create.
3. Click on SQL Database | Create.
4. Confirm Subscription and Resource Group; create a new Resource Group if required.
5. Enter Database name and Server; create a new Server if required.

The following screen is presented when creating a new database server.

6. Configure the SQL Database regarding:
   • Workload environment
   • Compute + storage
   • Backup storage redundancy

NOTE: the above configuration options will create a database with specifications relevant to development workloads. If your database must perform at greater levels configure the database accordingly.

7. Click on Next : Networking and then configure Networking options. The following is for illustrative purposes only. Consult an Azure administrator or subject matter expert for advice.

8. Configure Security, Additional settings, and Tags as required by your organization.
9. Check that the SQL Server and Databases are present in Azure Portal after creation.

For further information regarding creating Azure SQL Databases in Azure Portal, refer to the following Microsoft resource.

Quickstart: Create a single database – Azure SQL Database – https://docs.microsoft.com/en-us/azure/azure-sql/database/single-database-create-quickstart?view=azuresql&tabs=azure-portal

Create an “SDE Schema” Enterprise Geodatabase in Azure SQL database.

1. Connect to the Azure SQL Server Instance in Microsoft SQL Server Management Studio (SSMS).
2. Navigate to Databases > System Databases > master.
3. Right-click and select New Query.
4. Create an “sde” login.

CREATE LOGIN sde WITH PASSWORD = ‘Pa55w0rd1234’;

NOTE: Choose your own password for the “sde” login. The password included in the example above is not recommended for use in customer environments.

5. Create an “sde” database user.

CREATE USER sde FROM LOGIN sde;

6. Add “sde” to the LOGINMANAGER role.

ALTER ROLE loginmanager ADD MEMBER sde;

7. Navigate to Databases > databaseName.
8. Right-click and select New Query.
9. Create an “sde” user in the Azure SQL database.

CREATE USER sde FROM LOGIN sde;

10. Start ArcGIS Pro.
11. Use the Create Enterprise Geodatabase tool to create an enterprise geodatabase in the Azure SQL database.

NOTES:

• The Authorization File (typically named ‘keycodes’) will have to be copied to the Azure VM running ArcGIS Pro prior to running the Create Enterprise Geodatabase tool.
• The Database Administrator credentials (username and password) are the Server admin login values specified when creating the SQL Database Server.
• The Geodatabase Administrator Password becomes the password for the “sde” database user.

12. In SQL Server Management Studio, check that the geodatabase repository tables were created successfully.

Note: The list of tables above is not a complete list of the geodatabase repository tables.

13. Verify permissions for user ‘sde’ in the geodatabase.
    • Right-click the geodatabase and select Properties.
    • Click on Permissions.
    • Ensure the ‘sde’ user is selected. It should be granted the following permissions –
      • Connect
      • Create function
      • Create procedure
      • Create table
      • Create view
      • View database state
      • View definition

14. Check ALLOW_SNAPSHOT_ISOLATION, SNAPSHOT_ISOLATION_STATE_DESC and READ_COMMITTED_SNAPSHOT configuration for the geodatabase.

SELECT name, snapshot_isolation_state,
snapshot_isolation_state_desc,
is_read_committed_snapshot_on
FROM sys.databases
WHERE name = ‘DBName‘;

15. Check your results match the following.

16. Create a connection to the enterprise geodatabase as the geodatabase administrator (sde) in ArcGIS Pro.

For more information regarding geodatabase connections, refer to the following resource.

Connect to a database – https://pro.arcgis.com/en/pro-app/latest/help/projects/connect-to-a-database.htm

The long awaited GUI for managing ArcSDE administrative tasks such as versioning, connections and locks has arrived at 10.1!

The Geodatabase Administration Dialog can be accessed from ArcCatalog & ArcMap by right clicking the geodatabase connection.

Version Manager

The redesigned Version Manager has been moved into the Geodatabase Administration dialog box and includes the following new features:

• A tree view that lets you see how versions are related
• The ability to filter for a specific transactional or historical version, versions that could prevent the DEFAULT version from compressing to state 0, or versions owned by a specific user
• The ability to see all the properties of a selected version without having to open a separate dialog box
• A list recommending the order in which versions should be reconciled. Geodatabase administrators (sde/dbo) have access to the Reconcile Order subtab on the Geodatabase Administration dialog box. The subtab will only be available if you are signed in as the administrator.

More information can be found via the following web help link:

Version Properties:
http://resources.arcgis.com/en/help/main/10.1/index.html#//003n000000wm000000

Connections

You can use the Connections tab on the Geodatabase Administration dialog box to view all users connected to ArcSDE geodatabases. The information provided helps you monitor which users and client machines are connecting to your geodatabase and how long each session has been open.

You can disconnect user sessions from this dialog box by right-clicking a session and clicking Disconnect User.

In order to view ALL connections, you will need to be connected as the geodatabase administrator such as the sde/dbo user. The “Connections” tab is only visible, once a connection as the geodatabase administrator has been made. This will allow you to view both Direct and Application server connections.

More information can be found via the following web help link:
Viewing connected users in ArcGIS for Desktop
http://resources.arcgis.com/en/help/main/10.1/index.html#/Viewing_connected_user_sessions/019000000003000000/

Locks

The Locks tab allows you to view what type of locks are being held on data in an enterprise geodatabase and who is holding the lock. Through this dialog box, you can see active locks on datasets and versions and which user has acquired the lock.

In order to view ALL locks, you will need to be connected as the geodatabase administrator such as the sde/dbo user.

More information can be found via the following web help link:
Managing locks in the geodatabase
http://resources.arcgis.com/en/help/main/10.1/index.html#/Managing_locks_in_the_geodatabase/01900000000n000000/

Tania T.

An ArcGIS geodatabase is a collection of geographic datasets of various types held in a common file system folder, or a multiuser relational database management system such as IBM Db2, Microsoft SQL Server, Oracle, PostgreSQL, or SAP HANA.

Several cloud providers, such as Microsoft and Amazon, offer managed database services that are also supported by ArcGIS Enterprise, and you can create a geodatabase in these database service offerings as well.

Refer to the following resource for further information about the requirements for using ArcGIS Enterprise and ArcGIS Pro with databases in the cloud.

Requirements for using ArcGIS 11.0 and ArcGIS Pro 3.0 with databases in the cloud – https://enterprise.arcgis.com/en/system-requirements/latest/windows/databases-in-the-cloud.htm

There are two types of databases in the cloud: databases that are installed on virtual machines in the cloud, and databases that are available as database-as-a-service offerings. For database-as-a-service types, you do not have access to the machine that serves up the database and you do not have to perform maintenance tasks such as applying security patches on the database or operating system.

When talking about Microsoft Azure’s database-as-a-service offering, we use the term “Azure SQL Database”.

Azure SQL Database – https://azure.microsoft.com/en-au/products/azure-sql/database/#overview

Although Esri’s ArcGIS Enterprise Cloud Builder for Microsoft Azure (https://enterprise.arcgis.com/en/server/latest/cloud/azure/what-is-arcgis-server-cloud-builder-for-microsoft-azure.htm) can be used to add databases to an ArcGIS Server site on Microsoft Azure, in some circumstances clients may wish to manually create Enterprise Geodatabase in an Azure SQL Database with a geodatabase administrator named “sde”.

The geodatabase administrator in SQL Server – https://pro.arcgis.com/en/pro-app/latest/help/data/geodatabases/manage-sql-server/geodatabase-administrator-sqlserver.htm

A comparison of geodatabase owners in SQL Server – https://pro.arcgis.com/en/pro-app/latest/help/data/geodatabases/manage-sql-server/comparison-geodatabase-owners-sqlserver.htm

This blog article can be used as a reference for the workflows required to create an Azure SQL Database with an ”sde-owned” Enterprise Geodatabase.

Create an Azure SQL Database from Azure Portal.

1. Navigate to the relevant Resource Group.
2. Click on Create.
3. Click on SQL Database | Create.
4. Confirm Subscription and Resource Group; create a new Resource Group if required.
5. Enter Database name and Server; create a new Server if required.

The following screen is presented when creating a new database server.

6. Configure the SQL Database regarding:
   • Workload environment
   • Compute + storage
   • Backup storage redundancy

NOTE: the above configuration options will create a database with specifications relevant to development workloads. If your database must perform at greater levels configure the database accordingly.

7. Click on Next : Networking and then configure Networking options. The following is for illustrative purposes only. Consult an Azure administrator or subject matter expert for advice.

8. Configure Security, Additional settings, and Tags as required by your organization.
9. Check that the SQL Server and Databases are present in Azure Portal after creation.

For further information regarding creating Azure SQL Databases in Azure Portal, refer to the following Microsoft resource.

Quickstart: Create a single database – Azure SQL Database – https://docs.microsoft.com/en-us/azure/azure-sql/database/single-database-create-quickstart?view=azuresql&tabs=azure-portal

Create an “SDE Schema” Enterprise Geodatabase in Azure SQL database.

1. Connect to the Azure SQL Server Instance in Microsoft SQL Server Management Studio (SSMS).
2. Navigate to Databases > System Databases > master.
3. Right-click and select New Query.
4. Create an “sde” login.

CREATE LOGIN sde WITH PASSWORD = ‘Pa55w0rd1234’;

NOTE: Choose your own password for the “sde” login. The password included in the example above is not recommended for use in customer environments.

5. Create an “sde” database user.

CREATE USER sde FROM LOGIN sde;

6. Add “sde” to the LOGINMANAGER role.

ALTER ROLE loginmanager ADD MEMBER sde;

7. Navigate to Databases > databaseName.
8. Right-click and select New Query.
9. Create an “sde” user in the Azure SQL database.

CREATE USER sde FROM LOGIN sde;

10. Start ArcGIS Pro.
11. Use the Create Enterprise Geodatabase tool to create an enterprise geodatabase in the Azure SQL database.

NOTES:

• The Authorization File (typically named ‘keycodes’) will have to be copied to the Azure VM running ArcGIS Pro prior to running the Create Enterprise Geodatabase tool.
• The Database Administrator credentials (username and password) are the Server admin login values specified when creating the SQL Database Server.
• The Geodatabase Administrator Password becomes the password for the “sde” database user.

12. In SQL Server Management Studio, check that the geodatabase repository tables were created successfully.

Note: The list of tables above is not a complete list of the geodatabase repository tables.

13. Verify permissions for user ‘sde’ in the geodatabase.
    • Right-click the geodatabase and select Properties.
    • Click on Permissions.
    • Ensure the ‘sde’ user is selected. It should be granted the following permissions –
      • Connect
      • Create function
      • Create procedure
      • Create table
      • Create view
      • View database state
      • View definition

14. Check ALLOW_SNAPSHOT_ISOLATION, SNAPSHOT_ISOLATION_STATE_DESC and READ_COMMITTED_SNAPSHOT configuration for the geodatabase.

SELECT name, snapshot_isolation_state,
snapshot_isolation_state_desc,
is_read_committed_snapshot_on
FROM sys.databases
WHERE name = ‘DBName‘;

15. Check your results match the following.

16. Create a connection to the enterprise geodatabase as the geodatabase administrator (sde) in ArcGIS Pro.

For more information regarding geodatabase connections, refer to the following resource.

Connect to a database – https://pro.arcgis.com/en/pro-app/latest/help/projects/connect-to-a-database.htm