diff --git a/DESCRIPTION b/DESCRIPTION
index 9e59232..b3acef7 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -1,7 +1,7 @@
Package: eurlex
Type: Package
Title: Retrieve Data on European Union Law
-Version: 0.3.5
+Version: 0.3.6
Authors@R: c(person(given = "Michal",
family = "Ovadek",
role = c("aut", "cre", "cph"),
@@ -10,7 +10,6 @@ Authors@R: c(person(given = "Michal",
Description: Access to data on European Union laws and court decisions made easy with pre-defined 'SPARQL' queries and 'GET' requests.
License: GPL-3
Encoding: UTF-8
-LazyData: true
Depends:
R (>= 3.4.0)
Imports:
@@ -19,6 +18,7 @@ Imports:
xml2,
tidyr,
httr,
+ curl,
rvest,
rlang,
stringr,
diff --git a/NEWS.md b/NEWS.md
index 99b3bda..bdef761 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,3 +1,14 @@
+# eurlex 0.3.6
+
+## Major changes
+
+- `elx_run_query()` now fails gracefully in presence of internet/server problems
+- `elx_fetch_data()` now automatically fixes urls with parentheses (e.g. "32019H1115(01)" used to fail)
+
+## Minor changes
+
+- minor fixes to vignette
+
# eurlex 0.3.5
## Major changes
diff --git a/R/elx_curia_list.R b/R/elx_curia_list.R
index 0de6f4c..d8a7024 100644
--- a/R/elx_curia_list.R
+++ b/R/elx_curia_list.R
@@ -8,7 +8,8 @@
#' @param parse If `TRUE`, references to cases and appeals are parsed out from `case_info` into separate columns
#' @return
#' A data frame containing case identifiers and information as character columns. Where the case id
-#' contains a hyperlink to Eur-Lex, the CELEX identifier is retrieved as well.
+#' contains a hyperlink to Eur-Lex, the CELEX identifier is retrieved as well. Hyperlinks to Eur-Lex
+#' disappeared from more recent cases.
#' @importFrom rlang .data
#' @export
#' @examples
diff --git a/R/elx_fetch_data.R b/R/elx_fetch_data.R
index bd45ad8..f205e5e 100644
--- a/R/elx_fetch_data.R
+++ b/R/elx_fetch_data.R
@@ -22,6 +22,19 @@ elx_fetch_data <- function(url, type = c("title","text","ids"),
language <- paste(language_1,", ",language_2,";q=0.8, ",language_3,";q=0.7", sep = "")
+ if (stringr::str_detect(url,"celex")){
+
+ clx <- stringr::str_extract(url, "(?<=celex\\/).*") %>%
+ stringr::str_replace_all("\\(","%28") %>%
+ stringr::str_replace_all("\\)","%29") %>%
+ stringr::str_replace_all("\\/","%2F")
+
+ url <- paste("http://publications.europa.eu/resource/celex/",
+ clx,
+ sep = "")
+
+ }
+
if (type == "title"){
response <- httr::GET(url=url,
diff --git a/R/elx_run_query.R b/R/elx_run_query.R
index 277283f..18873f5 100644
--- a/R/elx_run_query.R
+++ b/R/elx_run_query.R
@@ -20,9 +20,7 @@ elx_run_query <- function(query = "", endpoint = "http://publications.europa.eu/
curlready <- paste(endpoint,"?query=",gsub("\\+","%2B", utils::URLencode(query, reserved = TRUE)), sep = "")
- sparql_response <- httr::GET(url = curlready,
- httr::add_headers('Accept' = 'application/sparql-results+xml')
- )
+ sparql_response <- graceful_http(curlready)
sparql_response_parsed <- sparql_response %>%
elx_parse_xml()
@@ -31,3 +29,49 @@ elx_run_query <- function(query = "", endpoint = "http://publications.europa.eu/
}
+#' Fail http call gracefully
+#'
+#' @importFrom rlang .data
+#'
+#' @noRd
+#'
+
+graceful_http <- function(remote_file) {
+
+ try_GET <- function(x, ...) {
+ tryCatch(
+ httr::GET(url = x,
+ #httr::timeout(1000000000),
+ httr::add_headers('Accept' = 'application/sparql-results+xml')),
+ error = function(e) conditionMessage(e),
+ warning = function(w) conditionMessage(w)
+ )
+ }
+
+ is_response <- function(x) {
+ class(x) == "response"
+ }
+
+ # First check internet connection
+ if (!curl::has_internet()) {
+ message("No internet connection.")
+ return(invisible(NULL))
+ }
+
+ # Then try for timeout problems
+ resp <- try_GET(remote_file)
+ if (!is_response(resp)) {
+ message(resp)
+ return(invisible(NULL))
+ }
+
+ # Then stop if status > 400
+ if (httr::http_error(resp)) {
+ httr::message_for_status(resp)
+ return(invisible(NULL))
+ }
+
+ return(resp)
+
+}
+
diff --git a/README.md b/README.md
index 51d079c..0781d5a 100644
--- a/README.md
+++ b/README.md
@@ -42,6 +42,12 @@ Please consider contributing to the maintanance and development of the package b
## Latest changes
+### eurlex 0.3.6
+
+- `elx_run_query()` now fails gracefully in presence of internet/server problems
+- `elx_fetch_data()` now automatically fixes urls with parentheses (e.g. "32019H1115(01)" used to fail)
+- minor fixes to vignette
+
### eurlex 0.3.5
- it is now possible to select all resource types available with `elx_make_query(resource_type = "any")`. Since there are nearly 1 million CELEX codes, use with discretion and expect long execution times
diff --git a/doc/eurlexpkg.html b/doc/eurlexpkg.html
index 495b333..5a049e3 100644
--- a/doc/eurlexpkg.html
+++ b/doc/eurlexpkg.html
@@ -453,14 +453,14 @@ elx_run_query(): Execute SPARQL queries
# elx_make_query("directive") %>%
# elx_run_query()
+#> # ... with 4,331 more rows
The function outputs a data.frame where each column corresponds to one of the requested variables, while the rows accumulate observations of the resource type satisfying the query criteria. Obviously, the more data is to be returned, the longer the execution time, varying from a few seconds to several minutes, depending also on your connection.
The first column always contains the unique URI of a “work” (legislative act or court judgment) which identifies each resource in Cellar. Several human-readable identifiers are normally associated with each “work” but the most useful one is CELEX, retrieved by default.
One column you should always pay attention to is type (as in resource_type). The URIs contained there reflect the FILTER argument in the SPARQL query, which is manually pre-specified. All resources are indexed as being of one type or another. For example, when retrieving directives, the results are going to return also delegated directives, which might not be desirable, depending on your needs. You can filter results by type to make the necessary adjustments. The queries are expansive by default in the spirit of erring on the side of over-inclusiveness rather than vice versa.
@@ -531,12 +531,12 @@ EuroVoc descriptors
select(celex, eurovoc, labels)
#> Joining, by = "eurovoc"
#> # A tibble: 10 x 3
-#> celex eurovoc labels
-#> <chr> <chr> <chr>
-#> 1 32012H00~ http://eurovoc.europa.eu~ informácie pre spotrebitela,vzdelávanie s~
-#> 2 31962H08~ http://eurovoc.europa.eu~ blahobyt
-#> 3 31974H04~ http://eurovoc.europa.eu~ Francúzska republika,Francúzsko
-#> 4 31996H05~ http://eurovoc.europa.eu~ formulár
+#> celex eurovoc labels
+#> <chr> <chr> <chr>
+#> 1 32012H0~ http://eurovoc.euro~ informácie pre spotrebitela,vzdelávanie spotreb~
+#> 2 31962H0~ http://eurovoc.euro~ blahobyt
+#> 3 31974H0~ http://eurovoc.euro~ Francúzska republika,Francúzsko
+#> 4 31996H0~ http://eurovoc.euro~ formulár
#> # ... with 6 more rows
@@ -581,7 +581,7 @@ Application
count(force) %>%
ggplot(aes(x = force, y = n)) +
geom_col()
-
+
Directives become naturally outdated with time. It might be all the more interesting to see which older acts are thus still surviving.
-
+
We want to know a bit more about the directives from 1970s that are still in force today. Their titles could give us a clue.
+#> # ... with 63 more rows
I will use the tidytext package to get a quick idea of what the legislation is about.
-
+
I use term-frequency inverse-document frequency (tf-idf) to weight the importance of the words in the wordcloud. If we used pure frequencies, the wordcloud would largely consist of words conveying little meaning (“the”, “and”, …).
This is an extremely basic application of the eurlex package. Much more sophisticated methods can be used to analyse both the content and metadata of European Union legislation. If the package is useful for your research, please consider citing the accompanying paper.
diff --git a/docs/404.html b/docs/404.html
index 21866c3..1b82b8d 100644
--- a/docs/404.html
+++ b/docs/404.html
@@ -79,7 +79,7 @@
eurlex
- 0.3.5
+ 0.3.6
diff --git a/docs/articles/eurlexpkg.html b/docs/articles/eurlexpkg.html
index dfc9641..97ce20e 100644
--- a/docs/articles/eurlexpkg.html
+++ b/docs/articles/eurlexpkg.html
@@ -39,7 +39,7 @@
eurlex
- 0.3.5
+ 0.3.6
@@ -249,14 +249,14 @@
# elx_make_query("directive") %>%
# elx_run_query()