Package Documentation¶
Scraped Lineups API Package.
Classes:
Name | Description |
---|---|
ScrapedLineupsAPI |
Represents Signal's Scraped Lineups API. |
ScrapedLineup |
Scraped Lineup. |
ScrapedLineup
dataclass
¶
Detailed information about a scraped lineup.
Attributes:
Name | Type | Description |
---|---|---|
lineup_id |
int
|
Integer. A unique identifier of the lineup line. |
message_id |
Optional[int]
|
Integer. A unique identifier of the message containing the specific lineup. A message can contain more than one lineup. |
external_message_id |
Optional[str]
|
String. It serves as a unique identifier for a message, supplied by any company that has integrated with Signal. |
parsed_part_id |
Optional[int]
|
Integer. A unique identifier for each email part. The email body and each attachment are considered different parsed parts. For an example the email body and its pdf attachment have same MessageID and different ParsedPartID. |
line_from |
Optional[int]
|
Nullable integer. The starting line from which the lineup is extracted. The email subject counts as line 0 and the body content starts from line 1. |
line_to |
Optional[int]
|
Nullable integer. The final line from which the lineup is extracted. For single line lineups LineFrom is equal to LineTo. For descriptive lineups that span across multiple lines we have LineTo>LineFrom. These two numbers help the user identify which part of the text has been used to extract the lineup data. |
in_line_order |
Optional[int]
|
Nullable integer. This integer is used to list different cargoes extracted from the same line. It is the case for lineups with multiple cargoes or buyers. A lineup with two cargoes like 'gasoil + fuel oil' is interpreted in our system as 2 lineups with same MessageID, same ParsedPartID, same LineNumber, same IMO, different LineupID and an incremental InLineOrder number. |
source |
Optional[str]
|
String. It describes the source of the information. Our system allows the user to inject data in many different ways, namely through email (Source='Email'), through Slack channels (Source='Slack') or through manual contributions directly from our frontend platform TSOP (Source='User'). |
updated_date |
Optional[datetime]
|
String, format YYYY-MM-DD HH:MM:SS, UTC timezone. Date on which the lineup has been reevaluated for the last time. In case of an email received by a broker one month ago and reprocessed through our engine today, this date will be today's. |
received_date |
Optional[datetime]
|
String, format YYYY-MM-DD HH:MM:SS, UTC timezone. Date on which the lineup has been injected into our system and processed. |
is_deleted |
Optional[bool]
|
Boolean. This value is true if the lineup is marked as Deleted. |
low_confidence |
Optional[bool]
|
Boolean. This value is true when the data extraction process does not return as output some fields that we believe to be more important than others in business terms. These fields are called critical fields. The value is true if at least one of the critical fields is missing.For example missing charterer or laycan. |
scraped_vessel_name |
Optional[str]
|
String. The vessel name as reported in the lineup line, i.e. 'Signal Alpha', 'Cpt A Stellatos', 'Genco Tiberius'. 'TBN' can also be found. |
scraped_imo |
Optional[str]
|
String. Vessel's IMO as reported in the lineup line, i.e. '9412036', '9439670', '9331555'. |
scraped_deadweight |
Optional[str]
|
String. The dead weight of the vessel as reported in the lineup line, i.e. '150249', '167'. |
scraped_year_built |
Optional[str]
|
String. The year built of the vessel as reported in the lineup line, i.e. '2004', '09'. |
imo |
Optional[int]
|
Integer. The seven-digits number that uniquely identifies the ship reported in the lineup. It is the result of our internally developed Vessel Mapper model. |
vessel_name |
Optional[str]
|
String. It is the current vessel name corresponding to the IMO mentioned above. Provided to better specify the vessel and its particulars. Source: our internal Vessel Database. |
deadweight |
Optional[int]
|
Integer. The dead weight in tonnes [t] corresponding to the IMO mentioned above. |
year_built |
Optional[int]
|
Integer, YYYY format. The year the vessel was built. Source: our internal Vessel Database. |
liquid_capacity |
Optional[int]
|
Integer, measured in cbm [cbm]. The liquid capacity of the IMO mentioned above. Source: our internal Vessel Database. |
vessel_type_id |
Optional[int]
|
Integer. Numeric ID corresponding to the different values of the VesselType field. 1-> Tanker, 3-> Dry, 4 -> Containers, 5 ->LNG (Liquified Natural gas), 6-> LPG (Liquified Petroleum Gas). |
vessel_type |
Optional[str]
|
String. Description of the type of the vessel (VesselTypeID), based on the carried cargo. Main categories are Tankers, Dry (bulk carriers), Containers, LNG and LPG. |
vessel_class_id |
Optional[int]
|
Integer. It is an ID corresponding to the different vessel classes of a certain vessel type, as split according to our internal Vessel Database. For example 84->VLCC, 85->Suezmax, 70->Capesize. |
vessel_class |
Optional[str]
|
String. Name of the vessel class the vessel belongs to. Assignment of a vessel to a certain VesselClass is based on the VesselType and the value of its Deadweight (if Tanker or Dry), its LiquidCap (if LNG/LPG) or its TEU (if Containers). For example, an Aframax is a Tanker vessel with Deadweight within the range 82kt - 125kt, while a Capesize is a Dry vessel with Deadweight within the range 120kt-220kt. LR2 are defined as Aframax, as only Deadweight is used to define vessel classes. |
commercial_operator_id |
Optional[int]
|
Integer. Numeric ID corresponding to the current maritime company that commercially manages the vessel corresponding to the IMO mentioned above. Source: Signal's proprietary algorithm. |
commercial_operator |
Optional[str]
|
String. Name of the current maritime company associated to CommercialOperatorID. Source: our internal Companies Database. |
scraped_eta |
Optional[str]
|
String. The estimated time of arrival of the lineup as reported in the original text. |
eta |
Optional[datetime]
|
Date, format YYYY-MM-DD. The mapped date corresponding to the estimated time of arrival date. |
scraped_etb |
Optional[str]
|
String. The estimated time of berthing of the lineup as reported in the original text. Strings tba (to be announced) and tbc (to be confirmed) can also be found. |
etb |
Optional[datetime]
|
Date, format YYYY-MM-DD. The mapped date corresponding to the estimated time of berthing date. |
scraped_etd |
Optional[str]
|
String. The estimated time of departure of the lineup as reported in the original text. Strings tba (to be announced) and tbc (to be confirmed) can also be found. |
etd |
Optional[datetime]
|
Date, format YYYY-MM-DD. The mapped date corresponding to the estimated time of departure date. |
scraped_location |
Optional[str]
|
String. The location reported in the original text of the lineup. Also, It could be reported in the subject of email. |
location_geo_id |
Optional[int]
|
Integer. It is the internal ID of the mapped location reported in the lineup. Our models convert and map a scraped string containing geo information to a specific entity of the Signal's proprietary geofencing structure. Each geo entity is identified by a taxonomy, an ID and a name. Examples: the string 'turkmenbashi' is mapped to a geo asset with name 'Turkmenbashi Refinery', ID 5086 and taxonomy GoeAsset (TaxonomyID=1); the string 'murmansk' is mapped to a geoasset with name 'Murmansk', ID 3761 and taxonomy Port (TaxonomyID=2); the string 'congo' is mapped to a geoasset with name 'Congo', ID 49 and taxonomy Country (TaxonomyID=3). |
location_name |
Optional[str]
|
String. The name of the Signal geo entity related to the reported location of the lineup. Examples: 'Qingdao', 'Nigeria', 'Yangpu', 'Quanzhou'. |
location_taxonomy_id |
Optional[int]
|
Integer. An internal ID corresponding to each taxonomy, the level of geo details, from 1 to 7. A terminal (geoasset) has the lowest taxonomy (TaxonomyID=1), a port has TaxonomyID=2, while countries and wider areas have higher taxonomy (TaxonomyID>=3). Examples of Level 0 areas (TaxonomyID=4) include 'Arabian Gulf', 'US Gulf' and 'East Mediterranean'. Level 1 areas (TaxonomyID=5) consist of multiple level 0 areas (TaxonomyID=4). For example, level 1 area 'Mediterranean' groups together the level 0 areas 'West Mediterranean', 'Central Mediterranean' and 'East Mediterranean'. Level 2 areas (TaxonomyID=6) consist of multiple level 1 areas (TaxonomyID=4). For example, level 2 area 'Mediterranean/UK Continent' groups together the 'Mediterranean' and 'UK Continent' level 1 areas. Level 3 areas (TaxonomyID=7) are the highest area grouping in our taxonomy and consist of multiple level 2 areas (TaxonomyID=6). Examples of such areas are 'Pacific America' or 'Africa'. These group together level 2 areas. For instance, 'Pacific America' groups together the level 2 areas West Coast North America', 'West Coast Mexico', 'West Coast Central America' and 'West Coast South America'. |
location_taxonomy |
Optional[str]
|
String. The extended name identifying the TaxonomyID. Possible values are: GeoAsset-> 1, Port -> 2, Country-> 3, Level0->4, Level1->5, Level2->6, Level3->7. |
operation_type_id |
Optional[int]
|
Interger. Numeric ID corresponding to the different values of the OperationType field. 1-> Loading, 2-> Discharge, 3 -> Waypoint, -2 -> NotSet, -1 -> Unknown. |
operation_type |
Optional[str]
|
String. Denotes the operation type of the lineup if explicitly mentioned, like 'loading' for load. |
scraped_quantity |
Optional[str]
|
String. The quantity as reported in the original text of the lineup. |
quantity |
Optional[float]
|
Integer. The mapped quantity measured in the corresponding value of QuantityUnit filed. |
quantity_unit |
Optional[str]
|
String. Denotes the quantity unit of the lineup. 1-> Tonnes, 2-> Barrels, -2 -> NotSet, -1 -> Unknown. |
scraped_cargo_type |
Optional[str]
|
String. The cargo type of the cargo as reported in the original text, often shortened. Examples: 'nhc', 'ulsd', 'ums', 'nap', 'go'. |
cargo_type_id |
Optional[int]
|
Integer. It is an internal ID corresponding to the mapped cargo type of the cargo. A proprietary model is responsible to match the reported cargo string to a specific cargo type in our hierarchy. Examples: 19-> Crude Oil, 16->Fueloil, 9-> Naphtha, 135-> Unleaded Motor Spirit, 12-> Gasoil, 60-> 'Ultra Low Sulphur Diesel (ULSD 10ppm)'. |
cargo_type |
Optional[str]
|
String. The extended name corresponding to the CargoTypeID field. Source: our internal CargoTypes database. |
cargo_group_id |
Optional[int]
|
Integer. Numeric ID corresponding to the high-level cargo type of the reported cargo type. It is provided to group cargoes and facilitate analytics. Examples: 130000->Dirty, 120000->Clean, 110000->IMO. |
cargo_group |
Optional[str]
|
String. The extended name corresponding to the CargoGroupID field. Source: our internal CargoTypes database. |
scraped_api_gravity |
Optional[str]
|
String. The API gravity reported in the original text on the lineup. |
api_gravity |
Optional[int]
|
Numeric. The mapped value corresponding to the API gravity. |
scraped_origin |
Optional[str]
|
String. The origin location reported in the original text of the lineup. |
origin_geo_id |
Optional[int]
|
Integer. An internal ID corresponding to the mapped origin location of the lineup. See LocationGeoID for more details. |
origin_name |
Optional[str]
|
String. The name of the Signal geo entity related to the reported origin location of the lineup. Examples: 'Fuel Pier', 'Port Of Arzew Terminal 2', 'CPC (Novorossiysk) SBM', 'Sheskharis Oil Terminal'. |
origin_taxonomy_id |
Optional[int]
|
Integer. An internal ID corresponding to the taxonomy of the mapped origin location. See LocationTaxonomyID for more details. |
origin_taxonomy |
Optional[str]
|
String. The extended name identifying the TaxonomyID. Possible values are: GeoAsset-> 1, Port -> 2, Country-> 3, Level0->4, Level1->5, Level2->6, Level3->7. |
scraped_destination |
Optional[str]
|
String. The destination location reported in the original text of the lineup. Strings tba (to be announced) and tbn (to be nominated) can also be found |
destination_geo_id |
Optional[int]
|
Integer. An internal ID corresponding to the mapped destination location of the lineup. See LocationGeoID for more details. |
destination_name |
Optional[str]
|
String. The name of the Signal geo entity related to the reported destination location of the lineup. Examples: 'Fuel Pier', 'Port Of Arzew Terminal 2', 'CPC (Novorossiysk) SBM', 'Sheskharis Oil Terminal'. |
destination_taxonomy_id |
Optional[int]
|
Integer. An internal ID corresponding to the taxonomy of the mapped destination location. See LocationTaxonomyID for more details. |
destination_taxonomy |
Optional[str]
|
String. The extended name identifying the TaxonomyID. Possible values are: GeoAsset-> 1, Port -> 2, Country-> 3, Level0->4, Level1->5, Level2->6, Level3->7. |
scraped_supplier |
Optional[str]
|
String. The supplier as reported in the original text of the lineup. |
supplier_id |
Optional[int]
|
Integer. Numeric ID corresponding to the supplier company that it is reported in the line. We use an internal mapper to find the correspondence between the reported string and our database. |
supplier |
Optional[str]
|
String. The company name corresponding to the SupplierID field. Provided to better specify the company involved in the business. Source: our internal Company Database. |
scraped_charterer |
Optional[str]
|
String. The charterer as reported in the original text of the lineup. |
charterer_id |
Optional[int]
|
Integer. Numeric ID corresponding to the chartering company that it is reported in the line. We use an internal mapper to find the correspondence between the reported string and our database. |
charterer |
Optional[str]
|
String. The company name corresponding to the ChartererID field. Provided to better specify the company involved in the business. Source: our internal Company Database. |
scraped_buyer |
Optional[str]
|
String. The buyer as reported in the original text of the lineup. |
buyer_id |
Optional[int]
|
Integer. Numeric ID corresponding to the buyer company that it is reported in the line. We use an internal mapper to find the correspondence between the reported string and our database. |
buyer |
Optional[str]
|
String. The company name corresponding to the BuyerID field. Provided to better specify the company involved in the business. Source: our internal Company Database. |
scraped_port_agent |
Optional[str]
|
String. The port agent as reported in the original text of the lineup. |
port_agent_id |
Optional[int]
|
Integer. Numeric ID corresponding to the port agent company that it is reported in the line. |
port_agent |
Optional[str]
|
String. The company name corresponding to the PortAgentID field. Provided to better specify the company involved in the business. Source: our internal Company Database. |
vessel_status_id |
Optional[int]
|
Integer. Numeric ID corresponding to the different values of the VesselStatus field. 1-> Expected, 2-> Arrived, 3-> At anchor, 4-> At berth, 5-> Sailed, 6-> Failed, 7-> Cancelled, 8-> Substituted, -2 -> NotSet, -1 -> Unknown. |
vessel_status |
Optional[str]
|
String denoting the vessel status of a lineup if explicitly mentioned, like 'expected for expected or 'waiting' for at anchor. |
content |
Optional[str]
|
String. The full content of the lineup. For a single line lineup it is the line content. For multi line lineups it is the collection of all the relevant parts of the text. |
subject |
Optional[str]
|
String. The email subject of the lineup. This field has content when Source='Email'. |
sender |
Optional[str]
|
String. Our own mapping of the shipping company sending out the market report through email. This string helps grouping emails sent by the same organization, but from different domains. It is often the case for big organizations operating worldwide. For example Sender= 'SSY' for both domains 'ssysin.com' and 'ssy.co'. |
is_private |
Optional[bool]
|
Boolean. A lineup is private if injected by a user into his own private account within TSOP. A user can provide private information through email forwarding or through Slack. Private lineup information stay in the account, are accessible by the account users only (people within the same company) and are the most valuable ones. |
Source code in signal_ocean/scraped_lineups/models.py
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 |
|
ScrapedLineupsAPI
¶
Bases: ScrapedDataAPI[ScrapedLineupsResponse, ScrapedLineup]
Represents Signal's Scraped Lineups API.
Source code in signal_ocean/scraped_lineups/scraped_lineups_api.py
16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 |
|
get_lineups(vessel_type, lineup_ids=None, message_ids=None, external_message_ids=None, received_date_from=None, received_date_to=None, updated_date_from=None, updated_date_to=None, imos=None, include_details=True, include_scraped_fields=True, include_vessel_details=True, include_labels=True, include_content=True, include_sender=True, include_debug_info=True)
¶
This function collects and returns the lineups by the given filters.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
vessel_type
|
int
|
Format - int32. Available values Tanker = 1, Dry = 3, Container = 4, Lng = 5, Lpg = 6 |
required |
lineup_ids
|
Optional[List[int]]
|
List - Comma separated list of LineupIDs |
None
|
message_ids
|
Optional[List[int]]
|
List - Comma separated list of MessageIDs |
None
|
external_message_ids
|
Optional[List[str]]
|
List - Comma separated list of ExternalMessageIDs |
None
|
received_date_from
|
Optional[datetime]
|
Format - date-time (as date-time in RFC3339). Earliest date the lineup received. Cannot be combined with 'Updated' dates |
None
|
received_date_to
|
Optional[datetime]
|
Format - date-time (as date-time in RFC3339). Latest date the lineup received. Cannot be combined with 'Updated' dates |
None
|
updated_date_from
|
Optional[datetime]
|
Format - date-time (as date-time in RFC3339). Earliest date the lineup updated. Cannot be combined with 'Received' dates |
None
|
updated_date_to
|
Optional[datetime]
|
Format - date-time (as date-time in RFC3339). Latest date the lineup updated. Cannot be combined with 'Received' dates |
None
|
imos
|
Optional[List[int]]
|
List - Comma separated list of IMOs |
None
|
include_details
|
Optional[bool]
|
Boolean - Whether to include additional lineup details in the response. |
True
|
include_scraped_fields
|
Optional[bool]
|
Boolean - Whether to include the relative scraped fields in the response. |
True
|
include_vessel_details
|
Optional[bool]
|
Boolean - Whether to include some vessel details in the response. |
True
|
include_labels
|
Optional[bool]
|
Boolean - Whether to include the relative labels in the response. |
True
|
include_content
|
Optional[bool]
|
Boolean - Whether to include the original message line (untouched) in the response. |
True
|
include_sender
|
Optional[bool]
|
Boolean - Whether to include some of the message sender details in the response. |
True
|
include_debug_info
|
Optional[bool]
|
Boolean - Whether to include some information about the distribution of the lineup in the response. |
True
|
Returns:
Type | Description |
---|---|
ScrapedLineup
|
An Iterable of ScrapedLineup objects, as we have defined in |
...
|
models.py Python file. |
Source code in signal_ocean/scraped_lineups/scraped_lineups_api.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
|
get_lineups_incremental(vessel_type, page_token=None, include_details=True, include_scraped_fields=True, include_vessel_details=True, include_labels=True, include_content=True, include_sender=True, include_debug_info=True)
¶
This function collects and returns lineups.
Specifically, all the lineups updated after the given page token. If page token is nullable, function will return all lineups.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
vessel_type
|
int
|
Format - int32. Available values Tanker = 1, Dry = 3, Container = 4, Lng = 5, Lpg = 6 |
required |
page_token
|
Optional[str]
|
String. The key that should be used as a parameter of the token to retrieve the relevant page. |
None
|
include_details
|
Optional[bool]
|
Boolean - Whether to include additional lineup details in the response. |
True
|
include_scraped_fields
|
Optional[bool]
|
Boolean - Whether to include the relative scraped fields in the response. |
True
|
include_vessel_details
|
Optional[bool]
|
Boolean - Whether to include some vessel details in the response. |
True
|
include_labels
|
Optional[bool]
|
Boolean - Whether to include the relative labels in the response. |
True
|
include_content
|
Optional[bool]
|
Boolean - Whether to include the original message line (untouched) in the response. |
True
|
include_sender
|
Optional[bool]
|
Boolean - Whether to include some of the message sender details in the response. |
True
|
include_debug_info
|
Optional[bool]
|
Boolean - Whether to include some information about the distribution of the lineup in the response. |
True
|
Returns:
Type | Description |
---|---|
IncrementalDataResponse[ScrapedLineup]
|
A dictionary containing a tuple of ScrapedLineup objects and |
IncrementalDataResponse[ScrapedLineup]
|
NextRequestToken. |
IncrementalDataResponse[ScrapedLineup]
|
ScrapedLineup object is defined in models.py Python file. |
IncrementalDataResponse[ScrapedLineup]
|
Next Request Token is used as page_token. |
Source code in signal_ocean/scraped_lineups/scraped_lineups_api.py
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 |
|
get_lineups_incremental_token(updated_date_from)
¶
This function returns a token to use in the incremental lineups endpoint.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
updated_date_from
|
datetime
|
Format - date-time (as date-time in RFC3339). Earliest date the cargo updated. Cannot be combined with 'Received' dates |
required |
Returns:
Type | Description |
---|---|
Optional[str]
|
A string containing the corresponding page token to |
Optional[str]
|
the provided datetime input. |
Source code in signal_ocean/scraped_lineups/scraped_lineups_api.py
156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 |
|