The EVEoj project is designed to push as much of the EVE logic and data into the browser as feasible in order to simplify the development of web tools for EVE. Code that doesn't NEED to run on a back-end server shouldn't HAVE to. It should be possible to write many tools with just HTML, JavaScript, and static hosting.
EVEoj tries to turn that possibility into reality.
For discussion and feedback, please use this EVE Online forum thread.
For issues and bugs, please refer to the GitHub issue tracker.
EVEoj requires jQuery.
Node.js support added in 0.2.x! Node.js functionality does not depend on jQuery, but JSON sources must be on the local file system.
The most recent version of the static data on the CDN is always available at:
https://cf.eve-oj.com/sdd/latest.
We all love Jita, but sometimes we'd like to visit (and shop!) in Amarr. Let's find out exactly how many jumps that trip will take with EVEoj in Node.js.
Coming at this as a Node.js developer instead of writing a web tool? This example
will display the route length from Jita to Amarr in Node.js.
Add the EVEoj package to your dependency list and download a current version of the JSON static data. Place the static data somewhere on a file-system accessible to your Node.js project.
var EVEoj = require("EVEoj"),
SDD = EVEoj.SDD.Create("json", {
path: "D:/static/sdd/201604290"
}),
map;
SDD.LoadMeta()
.then(function() {
map = EVEoj.map.Create(SDD, "K");
return map.Load();
})
.then(function() {
var jita = map.GetSystem({name: "Jita"});
var amarr = map.GetSystem({name: "Amarr"});
var route = map.Route(jita.ID, amarr.ID, [], false, false);
console.log("Jita to Amarr route length: " + route.length);
})
.caught(function(err) {
console.error(err);
});
Now let's get the item ID of something; perhaps so we can fetch some market data while we're busy shopping in Amarr? Sure, there are tools out there that can give us item ID's, but to build this functionality into OUR tool, we'd have to implement it from scratch. Let's try it out with EVEoj in the web browser.
This is the only time we'll talk to the server, and we're just grabbing static files, so any old web-server will do.
The first thing we need is a static data source. We create one with the 'json' source type and provide config settings to use the CDN hosted JSON (CORS-enabled) data.
var SDD = EVEoj.SDD.Create('json', {'path': 'https://cf.eve-oj.com/sdd/' + ver});
Once we have the SDD object, we need to load its metadata. This fetches information about the available tables and data. Since this is an asynchronous operation, it returns a chainable Promise object that is used to add success and failure handlers.
SDD.LoadMeta()
.then(function(arg){
return arg.source.GetTable('invTypes').Load();
})
.then(my_load_done_handler);
We use an anonymous success handler on the LoadMeta call to immediately turn around and get the "invTypes" table. This table is ~300KB of gzipped JSON data and should load quickly.
The table must be loaded asynchronously so it also uses Promises. In this case, we return the updated Promise chain, then add a custom handler called "my_load_done_handler" to be called when the table loading is successful.
function my_load_done_handler(arg) {
var name = $('#field_with_my_name').val(),
rxp = new RegExp('^' + name, 'i'),
tbl = arg.table,
results;
results = _.filter(tbl.data, function(entry) {
if (rxp.test(entry[tbl.c.typeName])) return true;
return false;
});
_.each(results, function(val, key, list) {
var typeName = val[tbl.c.typeName],
typeID = val[tbl.c.typeID];
// do something with typeName and typeID
});
}
Our table loading handler gets the name to search for from "somewhere". It then uses Underscore.js to filter the table data based on a regular expression test. We reference each entry's "typeName" data using the column mapping provided by the table (tbl.c). Finally, we use underscore again to iterate through our result set and do something with each matching result that was found.
While underscore is not a required dependency to use EVEoj, it does happen to be an extremely useful way to work with the data sets and interacting cleanly with underscore is a goal of the EVEoj core data model.
This project came about because I needed EVE's static data in several web tools I was writing (universe details, blueprint info, etc.) While exploring server-based methods for serving up this data, I realized that the ENTIRE static data export can be reduced to less than 50MB of (gzipped) JSON. Further, most of the data is not used at the same time; the relevant bits that you might use at any one time are routinely less than 500 KB (gzipped) in size. It seemed ridiculous to set up an entire back-end server stack to provide this data for something that could easily be served statically and loaded into a browser where JS can operate on it natively.
The EVEoj project is split into 3 layers. Each layer builds upon the next, and each is openly available so that you can integrate at the level that is most relevant for your application. Not at all like onion layers. More like the layers in a stack of tasty, tasty pancakes.
The first layer is an export of all the EVE static data (YAML and staticdata files) into a consistent JSON format with metainfo. Data is merged into "tables" that are more or less exactly what they sound like. However, since it's JSON, each entry in a table can contain more than just columns; complex data structures are also allowed.
The sde2json tool exports an EVE static data export to the Layer 1 format. While many individuals have helpfully provided SDE exports to different formats in the past, we are at their mercy for updates. In contrast, the sde2json tool is freely available on GitHub. Even if I vanish off the face of the planet, you'll always be able to get your SDE JSON exports by using that tool.
The second layer is a JavaScript library that asynchronously loads the static data tables into the browser. Large data sets (such as item details) are further segmented so they can be loaded piecemeal. Currently only an SDD table source is implemented, but I hope to eventually provide CREST and maybe even API sources at this level as well.
The EVEoj core exposes metainfo about tables and provides a consistent interface to access table data. This layer very specifically does NOT try to hide details of the data. Working with the table data exposed at this level is pretty close to working directly with the original EVE static data export.
The final layer is a set of addon JS modules that rely on the core but abstract away details of the underlying data to provide high-level interfaces. These modules work with concepts that should be familiar to any EVE player. Remember, all of this information is ALREADY available in the tables provided by core at Level 2; you just have to know the details to find it if you don't use these abstractions.
A major component of this project is to simplify and normalize access to the EVE static data without any need for a dynamic back-end server. The static data export produced by sde2json tallies up to about 48MB of gzipped JSON. Firefox can load and parse the entire data set in about 10s. The point of this project, however, is that you should never need to load the entire data set all at once.
The data is split intelligently (I hope) across tables to try and keep the actual working sizes down to around 500 KB or less. Very large data sets (e.g. invItems and celestial statistics) are further segmented to support partial loading. If you know the ID you are looking for, you can do a small partial load to get that entry without loading all 500,000 rows.
Part of what makes using static JSON appealing is the fact that the browser can cache these files so that a user rarely needs to hit the server to reload data. However, caching doesn't work as well if every tool uses its own, separately hosted version of the static data. Never mind that not every author has the capability to host 50MB of static JSON data in their environment.
To that end, I am committed to providing content-delivery network (CDN) hosted versions of the libraries and data. This will continue until I run out of money or regain my sanity. I currently provide hosting for other projects that total about a million hits a month, so I am not completely delusional as to what this entails.
You are free to use the links mentioned as "CDN" in the downloads section, both for the JS library and for the JSON data source of the SDD loader (CORS-enabled).
The CDN links for the JS library have both specific release versions as well as a ".x" version that will load the latest point release for a given version.
I cannot guarantee that the CDN hosted versions of these files will remain available forever. I will do everything possible to continue providing this service or to work with others to provide this service should it grow beyond my capacity. I of course reserve the right to cut you off at any time, without notice, for any reason, should I need to (both generally, everyone at once, or specifically, just you... yes, YOU... you know who you are).
Regardless, the value of this project is not diminished if the CDN hosted version disappears some day. All of the static data and JS libraries will continue to be made available; you will simply need to provide your own hosting for the data source used by your tools should that day arrive.
Tables are the basic data representation in EVEoj. They are more or less exactly what they sound like — a table of data indexed by a key — and are most literally just hash maps (e.g. JS objects). Tables in EVEoj have a few differences from a classic RDBMS table, owing to the fact that they are implemented in JSON and combine multiple data sources.
The specific tables available depends entirely on the data source. Let's take a look at the meta information for the tables provided in the CDN hosted JSON data.
This is the only time we'll talk to the server, and we're just grabbing static files, so any old web-server will do.
First, we need a static data source. We create one with the 'json' source type configured to use the CDN hosted JSON (CORS-enabled) data.
var SDD = EVEoj.SDD.Create('json', {'path': 'https://cf.eve-oj.com/sdd/' + ver});
Once we have the SDD object, we want its metadata, so we call LoadMeta. This fetches information about the available tables. Since this is an asynchronous operation, it returns a Promise object that is used to add success and failure handlers.
SDD.LoadMeta()
.then(function(arg){
var src = arg.source;
_.each(src.GetTables(), function(tableid){
var tbl = src.GetTable(tableid);
// tbl.name ; the table name/ID
// tbl.length ; the number of rows in the table
// tbl.segments.length ; the number of segments in the table
// tbl.columns ; an array containing the column names
// tbl.keyname ; the primary key for this table
// tbl.subkeys ; an array of subkeys for this table
// tbl.colmeta ; additional meta info about complex columns
});
});
We use an anonymous success handler on the LoadMeta call to immediately iterate through the tables available in this source. For each table there is a collection of metainfo that we can use for our very own nefarious, world-dominating purposes!
The main task of the EVEoj core is to enable the loading of tables from a data source. It parses metainfo about tables provided by the source it's pointed to, and uses this information to expose those tables and load their data as needed. It describes what columns are available, the primary key (and any sub keys), the number of rows, and how the data is segmented. It also tells the source where the table data can be found in the collection of static data files.
The table and metainfo data is generic and could be used to provide data beyond just the static data exports. For instance, putting your jump bridge network or user info into a JSON static file that conforms to this format would allow those tables to be loaded using the EVEoj core as well.
A specific goal of the EVEoj core layer and the tables it loads is that the data should work nicely with Underscore.js. While underscore is not a requirement to use EVEoj, it can definitely make working with the data much more friendly.
EVEoj.VERSION — the version of the libraryEVEoj.Const.M_per_LY — meters per light yearEVEoj.Const.M_per_AU — meters per AUEVEoj.Utils.FormatNum(number, decimals) → String
Returns a formatted string version of number with commas for thousands separators and a period for the decimal separator. Rounds to the number of decimals specified (use 0 to round to nearest whole integer).
Data sources for static data (as opposed to the theoretical CREST or API variants that are TOTALLY ON THE WAY I PROMISE! :)
The only current SDD data source implementation is the json type. However, this type supports both an AJAX implementation when running in the browser and a file-system implementation when running in Node.js. The Node.js fs implementation requires the JSON static data files to be stored locally.
EVEoj.SDD.Create(type[, config]) → EVEoj.SDD.Source
Creates a new EVEoj.SDD.Source object of the type specified. Accepts an optional config object with settings for the source type. Returns null if the specified type has no valid implementation.
This is the basic interface for all EVEoj.SDD data source implementations.
source.version — the version of the loaded source datasource.verdesc — an optional descriptive version string for the loaded source datasource.schema — the schema ID for the loaded source datasource.LoadMeta([opts]) → Promise
Begins asynchronous loading of the metadata for the source and immediately returns a Promise.
then_handlers(arg) — Receives the arg.source object that finished loading. The arg.context contains the optional opts.context provided to the original call.
source.HasTable(tableID) → Boolean
Tests if the source object contains the specific tableID. Will always return false until a successful LoadMeta has completed on the source.
source.GetTables() → Array<String>
Returns a list of the tableID's present in this source. Will always return an empty list until a successful LoadMeta has completed.
source.GetTable(tableID) → EVEoj.SDD.Source.Table
Returns the EVEoj.SDD.Source.Table object in this source with the tableID provided. Returns null if no such table exists in the source (or if LoadMeta has not yet completed).
This is the implementation of the json SDD source type. When creating this source type, the following settings can be provided as properties in a config object:
"json" (default) is only supported value
true
0
The table is the basic data access abstraction in the EVEoj core. Tables can only be retrieved from instantiated EVEoj.SDD.Source objects.
table.src — the EVEoj.SDD.Source that created this tabletable.name — the name/ID of this tabletable.keyname — the name of the primary key columntable.subkeys — an array of additional subkeys (or an empty array if none)table.columns — an array of column names for each entry in the tabletable.colmap — a reverse lookup hash mapping column names to column indexestable.c — a shorter name for table.colmap
table.colmeta — a hash (indexed by column name) of additional information for complex column structures; this data is informative only, and may not be accurate (or even exist) for any given columntable.length — the number of rows in this tabletable.loaded — the number of rows LOADED into this table (from 0 to table.length)table.segment — an array of segment info objects for the tabletable.data — the data object for this table, described in detail below
The table.data property is the primary means of interacting with the table data. It is a hash map indexed by the table.keyname. If no table.subkeys exist, then each entry in the hash map will be an Array containing the column values for that entry. The column values for each entry can be retrieved by index, and the index for a specific column name can be looked up in the table.colmap.
If table.subkeys DO exist, then each top-level entry in the table.data hash will not be an Array, but rather another hash map that is indexed by the first key in table.subkeys. Each of THOSE entries will ALSO be hash maps indexed by additional table.subkeys, nesting down until you run out of subkeys. The last entry in the nesting will then be an Array containing the column values for that entry.
Remember that each column value for an entry may itself be a complex JS Array or Object of its own with additional properties. Or, the value may be a simple base type. If a column contains a complex data type, the table.colmeta hash should hopefully contain an entry for that column with additional information about what to expect. This is purely optional and for human consumption only, and not always accurate (or even present).
table.Load([opts]) → Promise
Begins asynchronous loading of the table data and immediately returns a Promise. The optional opts.key property can be used to specify a key to look for. If a specific opts.key is provided, and the table has segments, ONLY the segment containing that key will be loaded. If no opts.key is provided then all table segments are loaded. An optional callback can be provided in opts.progress to receive progress updates for the load.
then_handlers(arg) — Receives the arg.table object that finished loading. The arg.context contains the optional opts.context provided to the original call.
progress_callback(arg) — The progress callback receives the arg.table object that is currently loading, the number of segments it arg.has finished loading, and the total number of segments it arg.needs to load (this is the total count required, including those segments already loaded). It also gets an arg.context from the optional opts.context provided to the original call. There is no guarantee that a progress callback will be called for each segment, or even that it will ever be called prior to the completion of the Load promise.
table.GetEntry(ID) → Array or Object
Returns the entry indexed by the ID provided, if it exists in the table. If the segment that contains this entry has NOT yet been loaded, it will return false. If the segment that contains this entry HAS been loaded and there is no matching entry, it will return null. Use strict equivalence tests to differentiate these scenarios.
The only reason to use this function rather than accessing table.data directly is to test for the existence vs. loaded distinction for an entry, as described above. Note that there is no way to know if an ID maps to a valid entry in the table until after the segment that should contain that ID has been loaded.
table.GetValue(ID, column) → column data
Returns the value for the named or indexed column (auto-determined based on use of a numeric or string value) from the entry in the table indexed by the ID provided. This is merely a convenience method for simple, single-value lookups. Note that this returns null or false as per the same logic in GetEntry whether the ID is loaded into the table or not. HOWEVER, there is no way to distinguish between those values and a valid entry with a column that actually contains a null or false value.
table.ColIter(column) → function(entry)
Returns an iterator function that accepts a single argument — the entry to process — and returns the value in the column specified (auto-index-detection based on whether a numeric or string is provided). This is a convenience method to get an iterator for the common use-case of applying Underscore functions to a column value on the table.data list (such as groupBy).
table.ColPred(column, comparison, value) → function(entry)
Returns a predicate function that accepts a single argument — the entry to process — and returns a boolean result by comparing the value in the column specified (auto-index-detection based on whether a numeric or string is provided) with the value provided. The type of comparison to use is provided as a string and can be ==, !=, ===, !==, <, <=, >, or >=. This is a convenience method to get a predicate for the common use-case of applying Underscore functions to a column value on the table.data list (such as filter).
A map provides an abstraction over regions, constellations, solar systems, and celestial info, as well as route planning, distance calculations, and more.
map.loaded — whether this map has been fully loaded or notEVEoj.map.Create(source, type[, config]) → EVEoj.map
Creates a new map object using the SDD data source provided. The map type should be one of 'K' (known space), 'X' (unknown/Jove space), or 'J' (wormhole space). An optional config object can also be provided with specific settings for this map.
All of these config options default to false.
config.jumps — include additional jump data (not needed for basic route calculations)config.planets — include planet celestial dataconfig.moons — moon celestial dataconfig.belts — belt celestial dataconfig.gates — jumpgate celestial dataconfig.stars — star celestial dataconfig.objects — celestial object reverse lookup (by solar system)config.landmarks — landmark dataAll of these config options default to false.
config.planets — include planet celestial dataconfig.moons — moon celestial dataconfig.stars — star celestial dataconfig.objects — celestial object reverse lookup (by solar system)map.Load([opts]) → Promise
Begins asynchronous loading of the map data from the source provided during object creation, then immediately returns a Promise. An optional callback can be provided in opts.progress to receive progress updates for the load.
then_handlers(arg) — Receives the arg.table object that finished loading. The arg.context contains the optional opts.context provided to the original call.
progress_callback(arg) — The progress callback receives the arg.map object that is currently loading, the number of segments it arg.has finished loading, and the total number of segments it arg.needs to load (this is the total count required, including those segments already loaded). It also gets an arg.context from the optional opts.context provided to the original call. There is no guarantee that a progress callback will be called for each segment, or even that it will ever be called prior to the completion of the Load promise.
map.GetSystem(system) → EVEoj.map.System
Gets an EVEoj.map.System object for the system provided (or null if no such system is found). The system provided may be either a numeric systemID # or a system name (case-sensitive).
map.GetSystems() → EVEoj.map.SystemIter
Gets a system iterator that lets you iterate through every system contained in the map.
map.JumpDist(fromSystemID, toSystemID) → Number
Gets the jump distance in light years between the fromSystemID and the toSystemID.
map.Route(fromSystemID, toSystemID, avoidList, avoidLow, avoidHi) → Array<systemID>
Gets an array containing the list of systemID's on the route connecting fromSystemID and toSystemID. A list of systemID's to avoid can be provided as an array to avoidList (use an empty array for no avoids). Note that this is a hard avoidance; if no route exists that does NOT contain a system in the avoidList, an empty route will be returned. The avoidLow and avoidHi values should be set to true or false as desired. Unlike the avoidList, these are soft avoids and valid routes will be returned even if it violates the avoidLow/Hi preference. Set both to false to simply find the shortest possible route.
system.ID — the system ID #system.name — the system namesystem.regionID — the region ID this system is insystem.constellationID — the constellation ID this system is insystem.pos.x|y|z — the x, y, and z coordinate for this systemsystem.posMax.x|y|z — the x, y, and z max coordinate for this systemsystem.posMin.x|y|z — the x, y, and z min coordinate for this systemsystem.border
system.fringe
system.corridor
system.hub
system.international
system.regional
system.constellation
system.contiguous — if this system is part of contiguous hisec (connected to Jita by hisec route)system.security — the true security value for the systemsystem.sec — the rounded/in-game security for the systemsystem.securityClass
system.wormholeClassID
system.stationCount — the number of stations in the systemsystem.jumps - an array of destination system IDs with gate jumps from this systemAn iterator for systems contained in a map.
iter.HasNext() → Boolean
Whether there are any more systems left in this iterator.
iter.Next() → EVEoj.map.System
Returns the next EVEoj.map.System object in the iterator.
Coming soon. Like... CCP "soon".
Coming soon. Partially implemented but not ported to new lib structure.
Coming really soon because... Crius.
This is a Node.js tool that turns an EVE static data export into JSON data. It also creates gzipped variants for ease of use with static hosting (those additional generators can be disabled).
This tool handles only the new full YAML and staticdata SDE, not the legacy SQL/DB format.
See the GitHub project for source and details.
usage: node --expose-gc index.js <--sde data_dir> [opts]
--sde data_dir — required: path to the EVE static data export files (the location where the .yaml, .staticdata, etc. files are located)--out sdd_dir — path of the output static data; defaults to {data_dir}/sdd--prefix prefix — specify a specific table space prefix to create (none specified == all)--gzip — set to 0 to disable gzip creation
This is project authored by me, nezroy.
For discussion and feedback, please use the EVE Online forum thread for this project.
For issues and bugs, please refer to the GitHub issue tracker.