I need to import an HTML content (a field in the MS SQL Server table) to a text file using scoop. Though I didn't get any error, imported field is expected. A few HTML tags are missing. Any help would be appreciated. Below is the sqoop comment
sqoop import --driver "net.sourceforge.jtds.jdbc.Driver" --connect \
"jdbc:jtds:sqlserver://abcd;instance=MSSQLSERVER;databaseName=abcd;domain=abcd.com" --username abcd -P \
--query "select * from abcd.html" --target-dir /user/abcd/abcd --split-by id --fields-terminated-by '|'
HTML Content Sample:
EMAIL SUBJECT: FW: The Update Site URL<br />
EMAIL BODY: <div>
<p style="margin: 0;"><span style="color:#1F497D">Hello. I will need assistance with this so the radio
can download calls to our server. Thank you.</span></p>
<p style="margin: 0;"><span style="color:#1F497D"></span></p>
<p style="margin: 0;"><span style="color:#1F497D"></span></p>
Related
I wrote a powershell script for setting downtimes in our monitoring software "checkmk" and now I have to convert that so a bash script because we are switching from windows server to linux.
user will be "XXX" password will be "YYY" and the hostname will be "ZZZ"
For that I used the documentation from CheckMK:
documentation for show sheduled downtimes
documentation for setting a downtime
and in the end it looks like that:
<#
.Synopsis
Downtime per Rest API bei Check MK Setzen
.DESCRIPTION
Doku https://apt-omd-vip.ads.vhv.de/umbrella/check_mk/openapi/#operation/cmk.gui.plugins.openapi.endpoints.downtime.create_host_related_downtime
30 min Downtime Setzen = 1800 sekunden
.EXAMPLE
Example of how to use this cmdlet
.EXAMPLE
Another example of how to use this cmdlet
#>
function set-Downtime
{
[CmdletBinding()]
[OutputType([int])]
Param
(
$USERNAME="XXX",
$PASSWORD="XXX",
[Parameter(Mandatory=$true)]
$HOSTNAME,
#Zeit soll die Downtime in Minuten angeben, wird dieseer Parameter leer gelassen wird Standdardmäßig 30min verwendet.
[Parameter(Mandatory=$false)]
[Int]$TIME=30
)
Begin
{
}
Process
{
#=============
# Abfrage der DownTimes, sonst kein Connect möglich
#=============
$headers = #{
‘Accept’ = ‘application/json’
‘Authorization’ = “Bearer $USERNAME $PASSWORD”
}
$body = #{
'host_name' = $HOSTNAME
}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12
$result = Invoke-RestMethod -uri “here was a URL” -Headers $headers -Body $body
#write-host($rest_uri)
#=============
# DownTime
#=============
$headers = #{
'Content-Type' = 'application/json'
‘Accept’ = ‘application/json’
‘Authorization’ = “Bearer $USERNAME $PASSWORD”
}
$datum_start = (get-date).AddHours(-2).ToString("u")
$datum_ende = (get-date).AddHours(-2).addminutes($TIME).ToString("u") ;
$comment = "$TIME min Restart Downtime, $EXTRAINFO"
$body =#{
start_time = $datum_start;
end_time = $datum_ende;
comment = $comment;
host_name = $HOSTNAME;
downtime_type = 'host';
} | ConvertTo-Json -Compress
$result = Invoke-RestMethod -Method Post -uri "here was a URL" -Headers $headers -Body $body
}
End
{
}
}
I apologize for the english-german mix in the script but in theory you shouldn't need my comments.
I reconstructed the command from this bash to powershell help blog
Here is one time my bash script hole and after that I will only post the part I changed:
#!/bin/bash
##Variables
base_url='here was a URL'
request_url="$base_url/all"
post_url="$base_url/host"
user="XXX"
passwd="YYY"
hostname="ZZZ"
user_pass="$user:$passwd"
#header=["Accep"t = "application/json"
#"Authorisation" = "Bearer $user $passwd"]
#body1=["host_name" = "ZZZ"]
curl -u $user_pass -X GET --header 'Accept: application/json' -d {'host_name: ZZZ'} 'here was an URL'
as I am still unexperienced in bash I tried my decleration of variables with the '' and "" quotes because I am still not sure when to use what (but thats not the question here)
Answer is following:
{"title": "You need to be authenticated to use the REST API.", "status": 401}
Next thing I tried was a header construct similar to my ps script:
header=( ["Accept"]="application/json" ["Authorization"]="Bearer XXX YYY")
curl --anyauth -X GET --header $header -d {'host_name: ZZZ'} 'here was an URL'
the answer this times was way longer but in the end I got the same error message:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "censored doc url">
<html>
<!-- FileName: index.html
Language: [en]
-->
<!--Head-->
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<meta http-equiv="X-UA-Compatible" content="IE=7" />
<title>McAfee Web Gateway - Notification</title>
<script src="censored" type="text/vascript" ></script>
<link rel="stylesheet" href="censored" />
</head>
<!--/Head-->
<!--Body-->
<body onload="swOnLoad();">
<table class='bodyTable'>
<tr>
<td class='bodyData' background='censored'>
<!--Logo-->
<table class='logoTable'>
<tr>
<td class='logoData'>
<a href='http://www.mcafee.com'>
<img src='censored'>a>
</td>
</tr>
</table>
<!--/Logo-->
<!--Contents-->
<!-- FileName: authenticationrequired.html
Language: [en]
-->
<!--Title-->
<table class='titleTable' background='censored.jpg'>
<tr>
<td class='titleData'>
Authentication Required
</td>
</tr>
</table>
<!--/Title-->
<!--Content-->
<table class="contentTable">
<tr>
<td class="contentData">
You must be authenticated to access this URL.
</td>
</tr>
</table>
<script language="javascript" type="text/javascript">
urlprotocol = "http";
statuscode=407;
if(statuscode==401 && urlprotocol == "ftp"){
document.write("<form name=\"ftpform\" method=\"get\" action=\"\">");
document.write("<table class=\"contentData\">");
document.write("<tr><td class=\"contentData\" colspan=2>Please enter youcredentials in the form below and click \"Access FTP\" button if your browser esn't present authentication prompt for FTP sites.</td></tr>");
document.write("<tr><td class=\"contentData\">Username:</td><td><input te=\"text\" id=\"ftpUsername\" name=\"ftpUsername\" size=40 /></td></tr>");
document.write("<tr><td class=\"contentData\">Password:</td><td><input te=\"password\" id=\"ftpPassword\" name=\"ftpPassword\" size=40 /></td></tr>");
document.write("<tr><td class=\"contentData\" colspan=2 align=center><int type=\"button\" onclick=\"redirectToFTP();\" value=\"Access FTP\" /></td></t");
document.write("</table>");
document.write("</form>");
}
function redirectToFTP(){
var username=escape(document.getElementById("ftpUsername").value);
var password=escape(document.getElementById("ftpPassword").value);
location.href = "ftp://"+username+":"+password+"#XXX:80/"
}
</script>
<!--/Content-->
<!--Info-->
<table class="infoTable">
<tr>
<td class="infoData">
<b>URL: </b><script type="censored");</script><br />
</td>
</tr>
</table>
<!--/Info-->
<!--/Contents-->
<!--Policy-->
<table class='policyTable'>
<tr>
<td class='policyHeading'>
<hr>
Company Acceptable Use Policy
</td>
</tr>
<tr>
<td class='policyData'>
This is an optional acceptable use disclaimer that appears on every pageYou may change the wording or remove this section entirely in index.html.
</td>
</tr>
</table>
<!--/Policy-->
<!--Foot-->
<table class='footTable'>
<tr>
<td class='helpDeskData' background='censored'>
For assistance, please contact your system administrator.
</td>
</tr>
<tr>
<td class='footData'>
generated <span id="time">2022-08-09 14:58:22</span> by McAfee Web Gatew
<br />
curl/7.60.0<br />
Node: censored<br />
Client IP: censored<br />
User: <br />
User-Groups: <br />
Authentication Method: <br />
Rule Set: Authentication with Kerberos and NTLM Fallback<br />
Rule: Perform Authentication<br />
</td>
</tr>
</table>
<!--/Foot-->
</td>
</tr>
</table>
</body>
<!--/Body-->
</html>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "censored">
<html>
<!-- FileName: index.html
Language: [en]
-->
<!--Head-->
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<meta http-equiv="X-UA-Compatible" content="IE=7" />
<title>McAfee Web Gateway - Notification</title>
<script src="censored" type="text/vascript" ></script>
<link rel="stylesheet" href="/mwg-internal/de5fs23hu73ds/files/default/styleeet.css" />
</head>
<!--/Head-->
<!--Body-->
<body onload="swOnLoad();">
<table class='bodyTable'>
<tr>
<td class='bodyData' background='censored'>
<!--Logo-->
<table class='logoTable'>
<tr>
<td class='logoData'>
<a href='http://www.mcafee.com'>
<img src='censored'>a>
</td>
</tr>
</table>
<!--/Logo-->
<!--Contents-->
<!-- FileName: authenticationrequired.html
Language: [en]
-->
<!--Title-->
<table class='titleTable' background='/mwg-internal/de5fs23hu73ds/files/defaulimg/bg_navbar.jpg'>
<tr>
<td class='titleData'>
Authentication Required
</td>
</tr>
</table>
<!--/Title-->
<!--Content-->
<table class="contentTable">
<tr>
<td class="contentData">
You must be authenticated to access this URL.
</td>
</tr>
</table>
<script language="javascript" type="text/javascript">
urlprotocol = "http";
statuscode=407;
if(statuscode==401 && urlprotocol == "ftp"){
document.write("<form name=\"ftpform\" method=\"get\" action=\"\">");
document.write("<table class=\"contentData\">");
document.write("<tr><td class=\"contentData\" colspan=2>Please enter youcredentials in the form below and click \"Access FTP\" button if your browser esn't present authentication prompt for FTP sites.</td></tr>");
document.write("<tr><td class=\"contentData\">Username:</td><td><input te=\"text\" id=\"ftpUsername\" name=\"ftpUsername\" size=40 /></td></tr>");
document.write("<tr><td class=\"contentData\">Password:</td><td><input te=\"password\" id=\"ftpPassword\" name=\"ftpPassword\" size=40 /></td></tr>");
document.write("<tr><td class=\"contentData\" colspan=2 align=center><int type=\"button\" onclick=\"redirectToFTP();\" value=\"Access FTP\" /></td></t");
document.write("</table>");
document.write("</form>");
}
function redirectToFTP(){
var username=escape(document.getElementById("ftpUsername").value);
var password=escape(document.getElementById("ftpPassword").value);
location.href = "ftp://"+username+":"+password+"#YYY:80/"
}
</script>
<!--/Content-->
<!--Info-->
<table class="infoTable">
<tr>
<td class="infoData">
<b>URL: </b><script type="text/javascript">break_line("http://setDownTim");</script><br />
</td>
</tr>
</table>
<!--/Info-->
<!--/Contents-->
<!--Policy-->
<table class='policyTable'>
<tr>
<td class='policyHeading'>
<hr>
Company Acceptable Use Policy
</td>
</tr>
<tr>
<td class='policyData'>
This is an optional acceptable use disclaimer that appears on every pageYou may change the wording or remove this section entirely in index.html.
</td>
</tr>
</table>
<!--/Policy-->
<!--Foot-->
<table class='footTable'>
<tr>
<td class='helpDeskData' background='/mwg-internal/de5fs23hu73ds/files/deflt/img/bg_navbar.jpg'>
For assistance, please contact your system administrator.
</td>
</tr>
<tr>
<td class='footData'>
generated <span id="time">2022-08-09 14:58:22</span> by McAfee Web Gatew
<br />
curl/7.60.0<br />
Node:censored<br />
Client IP: censored<br />
User: <br />
User-Groups: <br />
Authentication Method: <br />
Rule Set: Authentication with Kerberos and NTLM Fallback<br />
Rule: Perform Authentication<br />
</td>
</tr>
</table>
<!--/Foot-->
</td>
</tr>
</table>
</body>
<!--/Body-->
</html>
{"title": "You need to be authenticated to use the REST API.", "status": 401}
I needed to cut that part because I was over the characterlimit for the post...
I think that my error is not that hard to solve but I dont know enough about bash to search for the right things so please get easy on me. I used bash the first time in june this year so try to keep the answers a bit simpler than to an usual bash user please, thank you in advance for your help and consideration.
Like I guessed my problem was a minor one... only thing wrong was the ":" in my headers, just had to exchange them for "=".
Well got a new problem but I doubt that you guys can help me when I am not allowed to share more information about our server and applications.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
afer that message I did the same thing in the body like in my header but that didnt do the thing.
Here's a rewrite of your script in bash using curl, because it'll be easier to discuss against an example rather than an abstract in comments.
This, at least to my eyes, does exactly what your Powershell script is doing.
#!/bin/bash
url="https://YOUR-URL"
username="XXX"
password="YYY"
hostname="ZZZ"
# first Invoke-RestMethod, which is a GET with JSON...
# -1 means "use TLS1 or newer"
# -XGET is required here to force it to be a request
# that uses GET with a body, since curl would default to POST
# if '-d' is specified
# -w "%{http_code}\n" means output the http code
# if you want to suppress all other output and only
# get the http code output, specify these additional options:
# -o /dev/null -s
curl $url -1 -XGET \
-H "Accept: application/json" \
-H "Authorization: Bearer $username $password" \
-w "%{http_code}\n" \
-d "{ \"host_name\": \"$hostname\" }"
# Downtime Invoke-RestMethod
minutes_diff=30
start_diff=120
# because in your original, you do (now - 2 hours) + $minutes_diff
# so it's just now -(120 - $minutes_diff) minutes
end_diff=$(( 120 - $minutes_diff ))
# This was based on you using the .NET -u specifier which is
# intended for use with UTC DateTime objects
# I don't know if you actually want your timestamps derived
# from UTC time, which is what "-u" does for the date command
start_time=$(date -u -d "$start_diff minutes ago" +"%Y-%m-%d %H:%M:%SZ")
end_time=$(date -u -d "$end_diff minutes ago" +"%Y-%m-%d %H:%M:%SZ")
# you didn't have an EXTRAINFO in your original script
# so I left it out
comment="$minutes_diff min Restart Downtime"
# The -d #- <<-JSON uses a heredoc so I can
# more nicely express the JSON you send
curl $url -1 \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Authorization: Bearer $username $password" \
-w "%{http_code}\n" \
-d #- <<-JSON
{
"start_time": "$start_time",
"end_time": "$end_time",
"comment": "$comment",
"host_name": "$hostname",
"downtime_type": "host"
}
JSON
I have a html-template in a postgresql table, which I need to fetch and update from cmdline.
My idea was to use a Makefile to update the field.
I have figured, that I can use this command to fetch the file, using a function in a Makefile.
psql_conn=psql -U ${pg_user} -p $(pg_port_staging) -h $(pg_host) -d $(pg_database)
psql_shell=--tuples-only -P format=unaligned
fetch: ## fetch a given template id=XX
ifndef id
#echo "Missing id=XX"
#$(MAKE) list
#exit 1
else
$(call fetch_template,$(id),$(call fetch_name,$(id)))
endif
define fetch_template
$(shell echo "SELECT template FROM pdf_templates WHERE id = $(1)" | ${psql_conn} $(psql_shell} > $(2).html)
endef
define fetch_name
...
endef
However, I have a problem with inserting this template into the table again, after editing.
An example template file could be like this, so it would containts + { $ and UTF8 charaters.
<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org" xml:lang="en" lang="en">
<head>
<style type="text/css" th:inline="text">
body { margin: 0; font-family: Verdana, Geneva, sans-serif; font-size: 11px; }
a { text-decoration: none; }
</style>
</head
<body>
<div th:if="${hasAddresses}">
<h2>Adresser:</h2>
<article th:each="offerElement: ${offer.getElementsWithData().?[specification?.get('elementType') == 'fiber']}">
<div th:with="address=${offerElement.getParsedData().get()}">
<br /><span th:text="${address['deliveryStreet']}"></span> <span th:text="${address['deliveryHouseNumber']}"></span>
<span th:text="${address['deliveryFloor']}"></span> <span th:text="${address['deliveryDoor']}"></span>
<br /><span th:text="${address['deliveryZipCode']}"></span> <span th:text="${address['deliveryCity']}"></span>
<br /><span th:text="${address['deliveryCountry']}"></span>
</div>
</article>
</div>
</body>
</html>
It's a remote server, I don't have shell access to. So it's not an option to use pg_read_file
I have worked with the idea with using \lo_import File.html which upload the file to the server, but can't figure out how the content can be written into the table, and not just appear in the LargeObject table.
Another solution might be to use Ruby or Python, but I want to keep it as simple as possible.
The psql version i 9.4 however the server is running postgresql server 10.4.
Found a solution by converting the content to base64
define write_template
$(shell echo "UPDATE table SET template = convert_from(decode('$(shell base64 $(1))', 'base64'),'UTF8') WHERE name = '$(2)'" | ${psql_input})
endef
So take the content from base64 filename.html and use convert_from( decode($content , 'base64'),'UTF8') functions in postgresql to first decode the content to bytestring and thereafter convert it to UTF-8 format.
The ${psql_input} is just psql with user/host parameters.
I'm currently using Laravel4 for my project.
It's ok when I use this:
<iframe src="{{URL::to('/'). $video->path}}" width="500" height="281" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe>
But when I use this:
<iframe src="{{URL::to('/'). $video->path}}&autoplay=0" width="500" height="281" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe>
The iframe will return Symfony \ Component \ HttpKernel \ Exception \ NotFoundHttpException.
How can I fix this? Please help me out.
Thanks so much smile
I think you doing the query string wrong.
If the URL of the video is: yoursite.com/video1 and you append a &autoplay=0, then the url will be yoursite.com/video1&autoplay=0 which does not exist.
Change the &autoplay=0 to ?autoplay=0 to see if its works.
Although I'm getting correct options for an ajax-generated drop-down (based on the selection of another), I'm not seeing the value post to the PHP script. However, I see all the values from the normal HTML elements (not generated by ajax). What am I missing?
1. HTML produced by my scr_ajax.php script. - OK
$options is produced by the SQL query and the resulting selections are accurate. This is nearly identical to drop_down_1.).
<td>Drop Down 2</td>
<td></td>
<td>
<select name="drop_down_2" id="drop_down_2" value="" style="width:100%">
<option></option>
'.$options.'
</select>
</td>';
2. Where the ajax-generated HTML data goes.. - OK
Properly receives AJAX-generated form element above for the 2nd drop-down.):
...
<tr id="ajaxContent">
</tr>
...
3. Regular 'ol submit button.. - Not OK
drop_down_1 can be captured in $_POST data, but drop_down_2 cannot. I know I'm missing something here..)
<input type="submit" value="Submit Request" />
better start using jquery for everyday tasks.
http://api.jquery.com/jQuery.ajax/
[UPDATE: 11-25-2012]
I'm now able to combine the HTML and AJAX-generated post results by using the jQuery submit listener and populating the hidden field. However, this seems like more of a special technique/workaround as opposed to a more direct approach.
1st, I added the hidden input element:
<input type="hidden" name="drop_down_2" id="drop_down_2" value="" />
2nd, I added the jQuery submit listener:
<script type="text/javascript">
$(document).ready(function(){
$("#form_name").submit(function() {
var field = "drop_down_2";
var param = document.getElementById(field).options[document.getElementById(field).selectedIndex].value;
document.forms[0].elements[field].value = param;
});
});
</script>
[UPDATE: 12-01-2012]
It sounds like the proper solution involves mastery of the serialize function. I will post an update upon verification.
I'm trying to use cURL to post data to the form on this URL:
http://dq.sdc.bsnl.co.in/dq/reversePhone.seam?cid=812363
Seeing its source, the form looks like
<form id="revPhone" name="revPhone" method="post" action="/bsnl-web/reversePhone.seam;jsessionid=D238FA7A23A89A38C56B808B96F5D212" enctype="application/x-www-form-urlencoded" onkeyup="if (!check2(event)) {return false;};A4J.AJAX.Submit('loader2','revPhone',event,{'eventsQueue':'myqueue','parameters':{'revPhone:j_id16':'revPhone:j_id16'} ,'actionUrl':'/bsnl-web/reversePhone.seam;jsessionid=D238FA7A23A89A38C56B808B96F5D212','requestDelay':5} )">
<input type="hidden" name="revPhone" value="revPhone" />
<input type="hidden" name="revPhone:j_id12" />
<input id="revPhone:firstField" type="text" name="revPhone:firstField" maxlength="8" onkeydown="return removeEnter1(event)" />
<input id="revPhone:city" type="text" name="revPhone:city" value="Enter City Name" onblur="defaultText1();hideImage()" onfocus="defaultText2()" onkeyup="showImage(event)" />
Code pasted here: http://hastebin.com/wihunayilu.xml
Trying curl with these values:
curl --data "revPhone:firstField=24988872&revPhone:city=CHENNAI" http://dq.sdc.bsnl.co.in/dq/reversePhone.seam?cid=812363
I end up with the same page again as a response. How can I see (using firebug?) what parameters are passed to a post form so that I can correctly send the request to the server?
Thanks a lot
The form uses ajax to post to a URL which returns a page with a meta tag redirect. What I did to see the params was submit the form with the debugger open (I used chrome's dev tools, but this may also work in firebug), and then quickly press escape before the call finishes. The params look something like this:
AJAXREQUEST:loader2
revPhone:revPhone
revPhone:j_id12:
revPhone:firstField:24988872
revPhone:city:CHENNAI
revPhone:suggestionBoxId_selection:
javax.faces.ViewState:j_id4
revPhone:search:revPhone:search
and the response is:
<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta name="Ajax-Response" content="redirect" />
<meta name="Location" content="/bsnl-web/debug.seam;jsessionid=(some JSESSIONID)?cid=877632" />
</head></html>
(I edited out my jsessionid)
My curl line was:
curl --data "AJAXREQUEST=loader2&revPhone:suggestionBoxId_selection=&javax.faces.ViewState=j_id4&revPhone:search=search&revPhone=revPhone&revPhone:j_id12=&revPhone:firstField=24988872&revPhone:city=CHENNAI" http://dq.sdc.bsnl.co.in/dq/reversePhone.seam?cid=812363